diff options
Diffstat (limited to 'buildbot/docs/buildbot.html')
| -rw-r--r-- | buildbot/docs/buildbot.html | 9606 |
1 files changed, 0 insertions, 9606 deletions
diff --git a/buildbot/docs/buildbot.html b/buildbot/docs/buildbot.html deleted file mode 100644 index e4d1409..0000000 --- a/buildbot/docs/buildbot.html +++ /dev/null @@ -1,9606 +0,0 @@ -<html lang="en"> -<head> -<title>BuildBot Manual 0.7.10</title> -<meta http-equiv="Content-Type" content="text/html"> -<meta name="description" content="BuildBot Manual 0.7.10"> -<meta name="generator" content="makeinfo 4.11"> -<link title="Top" rel="top" href="#Top"> -<link href="http://www.gnu.org/software/texinfo/" rel="generator-home" title="Texinfo Homepage"> -<!-- -This is the BuildBot manual. - -Copyright (C) 2005,2006 Brian Warner - -Copying and distribution of this file, with or without -modification, are permitted in any medium without royalty -provided the copyright notice and this notice are preserved.--> -<meta http-equiv="Content-Style-Type" content="text/css"> -<style type="text/css"><!-- - pre.display { font-family:inherit } - pre.format { font-family:inherit } - pre.smalldisplay { font-family:inherit; font-size:smaller } - pre.smallformat { font-family:inherit; font-size:smaller } - pre.smallexample { font-size:smaller } - pre.smalllisp { font-size:smaller } - span.sc { font-variant:small-caps } - span.roman { font-family:serif; font-weight:normal; } - span.sansserif { font-family:sans-serif; font-weight:normal; } ---></style> -</head> -<body> -<h1 class="settitle">BuildBot Manual 0.7.10</h1> - <div class="contents"> -<h2>Table of Contents</h2> -<ul> -<li><a name="toc_Top" href="#Top">BuildBot</a> -<li><a name="toc_Introduction" href="#Introduction">1 Introduction</a> -<ul> -<li><a href="#History-and-Philosophy">1.1 History and Philosophy</a> -<li><a href="#System-Architecture">1.2 System Architecture</a> -<ul> -<li><a href="#BuildSlave-Connections">1.2.1 BuildSlave Connections</a> -<li><a href="#Buildmaster-Architecture">1.2.2 Buildmaster Architecture</a> -<li><a href="#Status-Delivery-Architecture">1.2.3 Status Delivery Architecture</a> -</li></ul> -<li><a href="#Control-Flow">1.3 Control Flow</a> -</li></ul> -<li><a name="toc_Installation" href="#Installation">2 Installation</a> -<ul> -<li><a href="#Requirements">2.1 Requirements</a> -<li><a href="#Installing-the-code">2.2 Installing the code</a> -<li><a href="#Creating-a-buildmaster">2.3 Creating a buildmaster</a> -<li><a href="#Upgrading-an-Existing-Buildmaster">2.4 Upgrading an Existing Buildmaster</a> -<li><a href="#Creating-a-buildslave">2.5 Creating a buildslave</a> -<ul> -<li><a href="#Buildslave-Options">2.5.1 Buildslave Options</a> -</li></ul> -<li><a href="#Launching-the-daemons">2.6 Launching the daemons</a> -<li><a href="#Logfiles">2.7 Logfiles</a> -<li><a href="#Shutdown">2.8 Shutdown</a> -<li><a href="#Maintenance">2.9 Maintenance</a> -<li><a href="#Troubleshooting">2.10 Troubleshooting</a> -<ul> -<li><a href="#Starting-the-buildslave">2.10.1 Starting the buildslave</a> -<li><a href="#Connecting-to-the-buildmaster">2.10.2 Connecting to the buildmaster</a> -<li><a href="#Forcing-Builds">2.10.3 Forcing Builds</a> -</li></ul> -</li></ul> -<li><a name="toc_Concepts" href="#Concepts">3 Concepts</a> -<ul> -<li><a href="#Version-Control-Systems">3.1 Version Control Systems</a> -<ul> -<li><a href="#Generalizing-VC-Systems">3.1.1 Generalizing VC Systems</a> -<li><a href="#Source-Tree-Specifications">3.1.2 Source Tree Specifications</a> -<li><a href="#How-Different-VC-Systems-Specify-Sources">3.1.3 How Different VC Systems Specify Sources</a> -<li><a href="#Attributes-of-Changes">3.1.4 Attributes of Changes</a> -</li></ul> -<li><a href="#Schedulers">3.2 Schedulers</a> -<li><a href="#BuildSet">3.3 BuildSet</a> -<li><a href="#BuildRequest">3.4 BuildRequest</a> -<li><a href="#Builder">3.5 Builder</a> -<li><a href="#Users">3.6 Users</a> -<ul> -<li><a href="#Doing-Things-With-Users">3.6.1 Doing Things With Users</a> -<li><a href="#Email-Addresses">3.6.2 Email Addresses</a> -<li><a href="#IRC-Nicknames">3.6.3 IRC Nicknames</a> -<li><a href="#Live-Status-Clients">3.6.4 Live Status Clients</a> -</li></ul> -<li><a href="#Build-Properties">3.7 Build Properties</a> -</li></ul> -<li><a name="toc_Configuration" href="#Configuration">4 Configuration</a> -<ul> -<li><a href="#Config-File-Format">4.1 Config File Format</a> -<li><a href="#Loading-the-Config-File">4.2 Loading the Config File</a> -<li><a href="#Testing-the-Config-File">4.3 Testing the Config File</a> -<li><a href="#Defining-the-Project">4.4 Defining the Project</a> -<li><a href="#Change-Sources-and-Schedulers">4.5 Change Sources and Schedulers</a> -<ul> -<li><a href="#Scheduler-Scheduler">4.5.1 Scheduler Scheduler</a> -<li><a href="#AnyBranchScheduler">4.5.2 AnyBranchScheduler</a> -<li><a href="#Dependent-Scheduler">4.5.3 Dependent Scheduler</a> -<li><a href="#Periodic-Scheduler">4.5.4 Periodic Scheduler</a> -<li><a href="#Nightly-Scheduler">4.5.5 Nightly Scheduler</a> -<li><a href="#Try-Schedulers">4.5.6 Try Schedulers</a> -<li><a href="#Triggerable-Scheduler">4.5.7 Triggerable Scheduler</a> -</li></ul> -<li><a href="#Merging-BuildRequests">4.6 Merging BuildRequests</a> -<li><a href="#Setting-the-slaveport">4.7 Setting the slaveport</a> -<li><a href="#Buildslave-Specifiers">4.8 Buildslave Specifiers</a> -<ul> -<li><a href="#When-Buildslaves-Go-Missing">4.8.1 When Buildslaves Go Missing</a> -</li></ul> -<li><a href="#On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves">4.9 On-Demand ("Latent") Buildslaves</a> -<ul> -<li><a href="#Amazon-Web-Services-Elastic-Compute-Cloud-_0028_0022AWS-EC2_0022_0029">4.9.1 Amazon Web Services Elastic Compute Cloud ("AWS EC2")</a> -<ul> -<li><a href="#Get-an-AWS-EC2-Account">4.9.1.1 Get an AWS EC2 Account</a> -<li><a href="#Create-an-AMI">4.9.1.2 Create an AMI</a> -<li><a href="#Configure-the-Master-with-an-EC2LatentBuildSlave">4.9.1.3 Configure the Master with an EC2LatentBuildSlave</a> -</li></ul> -<li><a href="#Dangers-with-Latent-Buildslaves">4.9.2 Dangers with Latent Buildslaves</a> -<li><a href="#Writing-New-Latent-Buildslaves">4.9.3 Writing New Latent Buildslaves</a> -</li></ul> -<li><a href="#Defining-Global-Properties">4.10 Defining Global Properties</a> -<li><a href="#Defining-Builders">4.11 Defining Builders</a> -<li><a href="#Defining-Status-Targets">4.12 Defining Status Targets</a> -<li><a href="#Debug-options">4.13 Debug options</a> -</li></ul> -<li><a name="toc_Getting-Source-Code-Changes" href="#Getting-Source-Code-Changes">5 Getting Source Code Changes</a> -<ul> -<li><a href="#Change-Sources">5.1 Change Sources</a> -<li><a href="#Choosing-ChangeSources">5.2 Choosing ChangeSources</a> -<li><a href="#CVSToys-_002d-PBService">5.3 CVSToys - PBService</a> -<li><a href="#Mail_002dparsing-ChangeSources">5.4 Mail-parsing ChangeSources</a> -<ul> -<li><a href="#Subscribing-the-Buildmaster">5.4.1 Subscribing the Buildmaster</a> -<li><a href="#Using-Maildirs">5.4.2 Using Maildirs</a> -<li><a href="#Parsing-Email-Change-Messages">5.4.3 Parsing Email Change Messages</a> -<ul> -<li><a href="#FCMaildirSource">5.4.3.1 FCMaildirSource</a> -<li><a href="#SyncmailMaildirSource">5.4.3.2 SyncmailMaildirSource</a> -<li><a href="#BonsaiMaildirSource">5.4.3.3 BonsaiMaildirSource</a> -<li><a href="#SVNCommitEmailMaildirSource">5.4.3.4 SVNCommitEmailMaildirSource</a> -</li></ul> -</li></ul> -<li><a href="#PBChangeSource">5.5 PBChangeSource</a> -<li><a href="#P4Source">5.6 P4Source</a> -<li><a href="#BonsaiPoller">5.7 BonsaiPoller</a> -<li><a href="#SVNPoller">5.8 SVNPoller</a> -<li><a href="#MercurialHook">5.9 MercurialHook</a> -<li><a href="#Bzr-Hook">5.10 Bzr Hook</a> -<li><a href="#Bzr-Poller">5.11 Bzr Poller</a> -</li></ul> -<li><a name="toc_Build-Process" href="#Build-Process">6 Build Process</a> -<ul> -<li><a href="#Build-Steps">6.1 Build Steps</a> -<ul> -<li><a href="#Common-Parameters">6.1.1 Common Parameters</a> -<li><a href="#Using-Build-Properties">6.1.2 Using Build Properties</a> -<li><a href="#Source-Checkout">6.1.3 Source Checkout</a> -<ul> -<li><a href="#CVS">6.1.3.1 CVS</a> -<li><a href="#SVN">6.1.3.2 SVN</a> -<li><a href="#Darcs">6.1.3.3 Darcs</a> -<li><a href="#Mercurial">6.1.3.4 Mercurial</a> -<li><a href="#Arch">6.1.3.5 Arch</a> -<li><a href="#Bazaar">6.1.3.6 Bazaar</a> -<li><a href="#Bzr">6.1.3.7 Bzr</a> -<li><a href="#P4">6.1.3.8 P4</a> -<li><a href="#Git">6.1.3.9 Git</a> -</li></ul> -<li><a href="#ShellCommand">6.1.4 ShellCommand</a> -<li><a href="#Simple-ShellCommand-Subclasses">6.1.5 Simple ShellCommand Subclasses</a> -<ul> -<li><a href="#Configure">6.1.5.1 Configure</a> -<li><a href="#Compile">6.1.5.2 Compile</a> -<li><a href="#Test">6.1.5.3 Test</a> -<li><a href="#TreeSize">6.1.5.4 TreeSize</a> -<li><a href="#PerlModuleTest">6.1.5.5 PerlModuleTest</a> -<li><a href="#SetProperty">6.1.5.6 SetProperty</a> -</li></ul> -<li><a href="#Python-BuildSteps">6.1.6 Python BuildSteps</a> -<ul> -<li><a href="#BuildEPYDoc">6.1.6.1 BuildEPYDoc</a> -<li><a href="#PyFlakes">6.1.6.2 PyFlakes</a> -<li><a href="#PyLint">6.1.6.3 PyLint</a> -</li></ul> -<li><a href="#Transferring-Files">6.1.7 Transferring Files</a> -<li><a href="#Steps-That-Run-on-the-Master">6.1.8 Steps That Run on the Master</a> -<li><a href="#Triggering-Schedulers">6.1.9 Triggering Schedulers</a> -<li><a href="#Writing-New-BuildSteps">6.1.10 Writing New BuildSteps</a> -<ul> -<li><a href="#Writing-BuildStep-Constructors">6.1.10.1 Writing BuildStep Constructors</a> -<li><a href="#BuildStep-LogFiles">6.1.10.2 BuildStep LogFiles</a> -<li><a href="#Reading-Logfiles">6.1.10.3 Reading Logfiles</a> -<li><a href="#Adding-LogObservers">6.1.10.4 Adding LogObservers</a> -<li><a href="#BuildStep-URLs">6.1.10.5 BuildStep URLs</a> -</li></ul> -</li></ul> -<li><a href="#Interlocks">6.2 Interlocks</a> -<li><a href="#Build-Factories">6.3 Build Factories</a> -<ul> -<li><a href="#BuildStep-Objects">6.3.1 BuildStep Objects</a> -<li><a href="#BuildFactory">6.3.2 BuildFactory</a> -<ul> -<li><a href="#BuildFactory-Attributes">6.3.2.1 BuildFactory Attributes</a> -<li><a href="#Quick-builds">6.3.2.2 Quick builds</a> -</li></ul> -<li><a href="#Process_002dSpecific-build-factories">6.3.3 Process-Specific build factories</a> -<ul> -<li><a href="#GNUAutoconf">6.3.3.1 GNUAutoconf</a> -<li><a href="#CPAN">6.3.3.2 CPAN</a> -<li><a href="#Python-distutils">6.3.3.3 Python distutils</a> -<li><a href="#Python_002fTwisted_002ftrial-projects">6.3.3.4 Python/Twisted/trial projects</a> -</li></ul> -</li></ul> -</li></ul> -<li><a name="toc_Status-Delivery" href="#Status-Delivery">7 Status Delivery</a> -<ul> -<li><a href="#WebStatus">7.1 WebStatus</a> -<ul> -<li><a href="#WebStatus-Configuration-Parameters">7.1.1 WebStatus Configuration Parameters</a> -<li><a href="#Buildbot-Web-Resources">7.1.2 Buildbot Web Resources</a> -<li><a href="#XMLRPC-server">7.1.3 XMLRPC server</a> -<li><a href="#HTML-Waterfall">7.1.4 HTML Waterfall</a> -</li></ul> -<li><a href="#MailNotifier">7.2 MailNotifier</a> -<li><a href="#IRC-Bot">7.3 IRC Bot</a> -<li><a href="#PBListener">7.4 PBListener</a> -<li><a href="#Writing-New-Status-Plugins">7.5 Writing New Status Plugins</a> -</li></ul> -<li><a name="toc_Command_002dline-tool" href="#Command_002dline-tool">8 Command-line tool</a> -<ul> -<li><a href="#Administrator-Tools">8.1 Administrator Tools</a> -<li><a href="#Developer-Tools">8.2 Developer Tools</a> -<ul> -<li><a href="#statuslog">8.2.1 statuslog</a> -<li><a href="#statusgui">8.2.2 statusgui</a> -<li><a href="#try">8.2.3 try</a> -<ul> -<li><a href="#try-_002d_002ddiff">8.2.3.1 try –diff</a> -</li></ul> -</li></ul> -<li><a href="#Other-Tools">8.3 Other Tools</a> -<ul> -<li><a href="#sendchange">8.3.1 sendchange</a> -<li><a href="#debugclient">8.3.2 debugclient</a> -</li></ul> -<li><a href="#_002ebuildbot-config-directory">8.4 .buildbot config directory</a> -</li></ul> -<li><a name="toc_Resources" href="#Resources">9 Resources</a> -<li><a name="toc_Developer_0027s-Appendix" href="#Developer_0027s-Appendix">Developer's Appendix</a> -<li><a name="toc_Index-of-Useful-Classes" href="#Index-of-Useful-Classes">Index of Useful Classes</a> -<li><a name="toc_Index-of-master_002ecfg-keys" href="#Index-of-master_002ecfg-keys">Index of master.cfg keys</a> -<li><a name="toc_Index" href="#Index">Index</a> -</li></ul> -</div> - - - -<div class="node"> -<p><hr> -<a name="Top"></a> -Next: <a rel="next" accesskey="n" href="#Introduction">Introduction</a>, -Previous: <a rel="previous" accesskey="p" href="#dir">(dir)</a>, -Up: <a rel="up" accesskey="u" href="#dir">(dir)</a> - -</div> - -<h2 class="unnumbered">BuildBot</h2> - -<p>This is the BuildBot manual. - - <p>Copyright (C) 2005,2006 Brian Warner - - <p>Copying and distribution of this file, with or without -modification, are permitted in any medium without royalty -provided the copyright notice and this notice are preserved. - -<ul class="menu"> -<li><a accesskey="1" href="#Introduction">Introduction</a>: What the BuildBot does. -<li><a accesskey="2" href="#Installation">Installation</a>: Creating a buildmaster and buildslaves, - running them. -<li><a accesskey="3" href="#Concepts">Concepts</a>: What goes on in the buildbot's little mind. -<li><a accesskey="4" href="#Configuration">Configuration</a>: Controlling the buildbot. -<li><a accesskey="5" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>: Discovering when to run a build. -<li><a accesskey="6" href="#Build-Process">Build Process</a>: Controlling how each build is run. -<li><a accesskey="7" href="#Status-Delivery">Status Delivery</a>: Telling the world about the build's results. -<li><a accesskey="8" href="#Command_002dline-tool">Command-line tool</a> -<li><a accesskey="9" href="#Resources">Resources</a>: Getting help. -<li><a href="#Developer_0027s-Appendix">Developer's Appendix</a> -<li><a href="#Index-of-Useful-Classes">Index of Useful Classes</a> -<li><a href="#Index-of-master_002ecfg-keys">Index of master.cfg keys</a> -<li><a href="#Index">Index</a>: Complete index. - -</li></ul> -<p>--- The Detailed Node Listing --- - -<p>Introduction - -</p> -<ul class="menu"> -<li><a href="#History-and-Philosophy">History and Philosophy</a> -<li><a href="#System-Architecture">System Architecture</a> -<li><a href="#Control-Flow">Control Flow</a> - -</li></ul> -<p>System Architecture - -</p> -<ul class="menu"> -<li><a href="#BuildSlave-Connections">BuildSlave Connections</a> -<li><a href="#Buildmaster-Architecture">Buildmaster Architecture</a> -<li><a href="#Status-Delivery-Architecture">Status Delivery Architecture</a> - -</li></ul> -<p>Installation - -</p> -<ul class="menu"> -<li><a href="#Requirements">Requirements</a> -<li><a href="#Installing-the-code">Installing the code</a> -<li><a href="#Creating-a-buildmaster">Creating a buildmaster</a> -<li><a href="#Upgrading-an-Existing-Buildmaster">Upgrading an Existing Buildmaster</a> -<li><a href="#Creating-a-buildslave">Creating a buildslave</a> -<li><a href="#Launching-the-daemons">Launching the daemons</a> -<li><a href="#Logfiles">Logfiles</a> -<li><a href="#Shutdown">Shutdown</a> -<li><a href="#Maintenance">Maintenance</a> -<li><a href="#Troubleshooting">Troubleshooting</a> - -</li></ul> -<p>Creating a buildslave - -</p> -<ul class="menu"> -<li><a href="#Buildslave-Options">Buildslave Options</a> - -</li></ul> -<p>Troubleshooting - -</p> -<ul class="menu"> -<li><a href="#Starting-the-buildslave">Starting the buildslave</a> -<li><a href="#Connecting-to-the-buildmaster">Connecting to the buildmaster</a> -<li><a href="#Forcing-Builds">Forcing Builds</a> - -</li></ul> -<p>Concepts - -</p> -<ul class="menu"> -<li><a href="#Version-Control-Systems">Version Control Systems</a> -<li><a href="#Schedulers">Schedulers</a> -<li><a href="#BuildSet">BuildSet</a> -<li><a href="#BuildRequest">BuildRequest</a> -<li><a href="#Builder">Builder</a> -<li><a href="#Users">Users</a> -<li><a href="#Build-Properties">Build Properties</a> - -</li></ul> -<p>Version Control Systems - -</p> -<ul class="menu"> -<li><a href="#Generalizing-VC-Systems">Generalizing VC Systems</a> -<li><a href="#Source-Tree-Specifications">Source Tree Specifications</a> -<li><a href="#How-Different-VC-Systems-Specify-Sources">How Different VC Systems Specify Sources</a> -<li><a href="#Attributes-of-Changes">Attributes of Changes</a> - -</li></ul> -<p>Users - -</p> -<ul class="menu"> -<li><a href="#Doing-Things-With-Users">Doing Things With Users</a> -<li><a href="#Email-Addresses">Email Addresses</a> -<li><a href="#IRC-Nicknames">IRC Nicknames</a> -<li><a href="#Live-Status-Clients">Live Status Clients</a> - -</li></ul> -<p>Configuration - -</p> -<ul class="menu"> -<li><a href="#Config-File-Format">Config File Format</a> -<li><a href="#Loading-the-Config-File">Loading the Config File</a> -<li><a href="#Testing-the-Config-File">Testing the Config File</a> -<li><a href="#Defining-the-Project">Defining the Project</a> -<li><a href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> -<li><a href="#Setting-the-slaveport">Setting the slaveport</a> -<li><a href="#Buildslave-Specifiers">Buildslave Specifiers</a> -<li><a href="#On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves">On-Demand ("Latent") Buildslaves</a> -<li><a href="#Defining-Global-Properties">Defining Global Properties</a> -<li><a href="#Defining-Builders">Defining Builders</a> -<li><a href="#Defining-Status-Targets">Defining Status Targets</a> -<li><a href="#Debug-options">Debug options</a> - -</li></ul> -<p>Change Sources and Schedulers - -</p> -<ul class="menu"> -<li><a href="#Scheduler-Scheduler">Scheduler Scheduler</a> -<li><a href="#AnyBranchScheduler">AnyBranchScheduler</a> -<li><a href="#Dependent-Scheduler">Dependent Scheduler</a> -<li><a href="#Periodic-Scheduler">Periodic Scheduler</a> -<li><a href="#Nightly-Scheduler">Nightly Scheduler</a> -<li><a href="#Try-Schedulers">Try Schedulers</a> -<li><a href="#Triggerable-Scheduler">Triggerable Scheduler</a> - -</li></ul> -<p>Buildslave Specifiers -</p> -<ul class="menu"> -<li><a href="#When-Buildslaves-Go-Missing">When Buildslaves Go Missing</a> - -</li></ul> -<p>On-Demand ("Latent") Buildslaves -</p> -<ul class="menu"> -<li><a href="#Amazon-Web-Services-Elastic-Compute-Cloud-_0028_0022AWS-EC2_0022_0029">Amazon Web Services Elastic Compute Cloud ("AWS EC2")</a> -<li><a href="#Dangers-with-Latent-Buildslaves">Dangers with Latent Buildslaves</a> -<li><a href="#Writing-New-Latent-Buildslaves">Writing New Latent Buildslaves</a> - -</li></ul> -<p>Getting Source Code Changes - -</p> -<ul class="menu"> -<li><a href="#Change-Sources">Change Sources</a> -<li><a href="#Choosing-ChangeSources">Choosing ChangeSources</a> -<li><a href="#CVSToys-_002d-PBService">CVSToys - PBService</a> -<li><a href="#Mail_002dparsing-ChangeSources">Mail-parsing ChangeSources</a> -<li><a href="#PBChangeSource">PBChangeSource</a> -<li><a href="#P4Source">P4Source</a> -<li><a href="#BonsaiPoller">BonsaiPoller</a> -<li><a href="#SVNPoller">SVNPoller</a> -<li><a href="#MercurialHook">MercurialHook</a> -<li><a href="#Bzr-Hook">Bzr Hook</a> -<li><a href="#Bzr-Poller">Bzr Poller</a> - -</li></ul> -<p>Mail-parsing ChangeSources - -</p> -<ul class="menu"> -<li><a href="#Subscribing-the-Buildmaster">Subscribing the Buildmaster</a> -<li><a href="#Using-Maildirs">Using Maildirs</a> -<li><a href="#Parsing-Email-Change-Messages">Parsing Email Change Messages</a> - -</li></ul> -<p>Parsing Email Change Messages - -</p> -<ul class="menu"> -<li><a href="#FCMaildirSource">FCMaildirSource</a> -<li><a href="#SyncmailMaildirSource">SyncmailMaildirSource</a> -<li><a href="#BonsaiMaildirSource">BonsaiMaildirSource</a> -<li><a href="#SVNCommitEmailMaildirSource">SVNCommitEmailMaildirSource</a> - -</li></ul> -<p>Build Process - -</p> -<ul class="menu"> -<li><a href="#Build-Steps">Build Steps</a> -<li><a href="#Interlocks">Interlocks</a> -<li><a href="#Build-Factories">Build Factories</a> - -</li></ul> -<p>Build Steps - -</p> -<ul class="menu"> -<li><a href="#Common-Parameters">Common Parameters</a> -<li><a href="#Using-Build-Properties">Using Build Properties</a> -<li><a href="#Source-Checkout">Source Checkout</a> -<li><a href="#ShellCommand">ShellCommand</a> -<li><a href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a> -<li><a href="#Python-BuildSteps">Python BuildSteps</a> -<li><a href="#Transferring-Files">Transferring Files</a> -<li><a href="#Steps-That-Run-on-the-Master">Steps That Run on the Master</a> -<li><a href="#Triggering-Schedulers">Triggering Schedulers</a> -<li><a href="#Writing-New-BuildSteps">Writing New BuildSteps</a> - -</li></ul> -<p>Source Checkout - -</p> -<ul class="menu"> -<li><a href="#CVS">CVS</a> -<li><a href="#SVN">SVN</a> -<li><a href="#Darcs">Darcs</a> -<li><a href="#Mercurial">Mercurial</a> -<li><a href="#Arch">Arch</a> -<li><a href="#Bazaar">Bazaar</a> -<li><a href="#Bzr">Bzr</a> -<li><a href="#P4">P4</a> -<li><a href="#Git">Git</a> - -</li></ul> -<p>Simple ShellCommand Subclasses - -</p> -<ul class="menu"> -<li><a href="#Configure">Configure</a> -<li><a href="#Compile">Compile</a> -<li><a href="#Test">Test</a> -<li><a href="#TreeSize">TreeSize</a> -<li><a href="#PerlModuleTest">PerlModuleTest</a> -<li><a href="#SetProperty">SetProperty</a> - -</li></ul> -<p>Python BuildSteps - -</p> -<ul class="menu"> -<li><a href="#BuildEPYDoc">BuildEPYDoc</a> -<li><a href="#PyFlakes">PyFlakes</a> -<li><a href="#PyLint">PyLint</a> - -</li></ul> -<p>Writing New BuildSteps - -</p> -<ul class="menu"> -<li><a href="#BuildStep-LogFiles">BuildStep LogFiles</a> -<li><a href="#Reading-Logfiles">Reading Logfiles</a> -<li><a href="#Adding-LogObservers">Adding LogObservers</a> -<li><a href="#BuildStep-URLs">BuildStep URLs</a> - -</li></ul> -<p>Build Factories - -</p> -<ul class="menu"> -<li><a href="#BuildStep-Objects">BuildStep Objects</a> -<li><a href="#BuildFactory">BuildFactory</a> -<li><a href="#Process_002dSpecific-build-factories">Process-Specific build factories</a> - -</li></ul> -<p>BuildStep Objects - -</p> -<ul class="menu"> -<li><a href="#BuildFactory-Attributes">BuildFactory Attributes</a> -<li><a href="#Quick-builds">Quick builds</a> - -</li></ul> -<p>BuildFactory - -</p> -<ul class="menu"> -<li><a href="#BuildFactory-Attributes">BuildFactory Attributes</a> -<li><a href="#Quick-builds">Quick builds</a> - -</li></ul> -<p>Process-Specific build factories - -</p> -<ul class="menu"> -<li><a href="#GNUAutoconf">GNUAutoconf</a> -<li><a href="#CPAN">CPAN</a> -<li><a href="#Python-distutils">Python distutils</a> -<li><a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a> - -</li></ul> -<p>Status Delivery - -</p> -<ul class="menu"> -<li><a href="#WebStatus">WebStatus</a> -<li><a href="#MailNotifier">MailNotifier</a> -<li><a href="#IRC-Bot">IRC Bot</a> -<li><a href="#PBListener">PBListener</a> -<li><a href="#Writing-New-Status-Plugins">Writing New Status Plugins</a> - -</li></ul> -<p>WebStatus - -</p> -<ul class="menu"> -<li><a href="#WebStatus-Configuration-Parameters">WebStatus Configuration Parameters</a> -<li><a href="#Buildbot-Web-Resources">Buildbot Web Resources</a> -<li><a href="#XMLRPC-server">XMLRPC server</a> -<li><a href="#HTML-Waterfall">HTML Waterfall</a> - -</li></ul> -<p>Command-line tool - -</p> -<ul class="menu"> -<li><a href="#Administrator-Tools">Administrator Tools</a> -<li><a href="#Developer-Tools">Developer Tools</a> -<li><a href="#Other-Tools">Other Tools</a> -<li><a href="#g_t_002ebuildbot-config-directory">.buildbot config directory</a> - -</li></ul> -<p>Developer Tools - -</p> -<ul class="menu"> -<li><a href="#statuslog">statuslog</a> -<li><a href="#statusgui">statusgui</a> -<li><a href="#try">try</a> - -</li></ul> -<p>waiting for results - -</p> -<ul class="menu"> -<li><a href="#try-_002d_002ddiff">try --diff</a> - -</li></ul> -<p>Other Tools - -</p> -<ul class="menu"> -<li><a href="#sendchange">sendchange</a> -<li><a href="#debugclient">debugclient</a> - - </ul> - -<div class="node"> -<p><hr> -<a name="Introduction"></a> -Next: <a rel="next" accesskey="n" href="#Installation">Installation</a>, -Previous: <a rel="previous" accesskey="p" href="#Top">Top</a>, -Up: <a rel="up" accesskey="u" href="#Top">Top</a> - -</div> - -<h2 class="chapter">1 Introduction</h2> - -<p><a name="index-introduction-1"></a> -The BuildBot is a system to automate the compile/test cycle required by most -software projects to validate code changes. By automatically rebuilding and -testing the tree each time something has changed, build problems are -pinpointed quickly, before other developers are inconvenienced by the -failure. The guilty developer can be identified and harassed without human -intervention. By running the builds on a variety of platforms, developers -who do not have the facilities to test their changes everywhere before -checkin will at least know shortly afterwards whether they have broken the -build or not. Warning counts, lint checks, image size, compile time, and -other build parameters can be tracked over time, are more visible, and -are therefore easier to improve. - - <p>The overall goal is to reduce tree breakage and provide a platform to -run tests or code-quality checks that are too annoying or pedantic for -any human to waste their time with. Developers get immediate (and -potentially public) feedback about their changes, encouraging them to -be more careful about testing before checkin. - - <p>Features: - - <ul> -<li>run builds on a variety of slave platforms -<li>arbitrary build process: handles projects using C, Python, whatever -<li>minimal host requirements: python and Twisted -<li>slaves can be behind a firewall if they can still do checkout -<li>status delivery through web page, email, IRC, other protocols -<li>track builds in progress, provide estimated completion time -<li>flexible configuration by subclassing generic build process classes -<li>debug tools to force a new build, submit fake Changes, query slave status -<li>released under the GPL -</ul> - -<ul class="menu"> -<li><a accesskey="1" href="#History-and-Philosophy">History and Philosophy</a> -<li><a accesskey="2" href="#System-Architecture">System Architecture</a> -<li><a accesskey="3" href="#Control-Flow">Control Flow</a> -</ul> - -<div class="node"> -<p><hr> -<a name="History-and-Philosophy"></a> -Next: <a rel="next" accesskey="n" href="#System-Architecture">System Architecture</a>, -Previous: <a rel="previous" accesskey="p" href="#Introduction">Introduction</a>, -Up: <a rel="up" accesskey="u" href="#Introduction">Introduction</a> - -</div> - -<h3 class="section">1.1 History and Philosophy</h3> - -<p><a name="index-Philosophy-of-operation-2"></a> -The Buildbot was inspired by a similar project built for a development -team writing a cross-platform embedded system. The various components -of the project were supposed to compile and run on several flavors of -unix (linux, solaris, BSD), but individual developers had their own -preferences and tended to stick to a single platform. From time to -time, incompatibilities would sneak in (some unix platforms want to -use <code>string.h</code>, some prefer <code>strings.h</code>), and then the tree -would compile for some developers but not others. The buildbot was -written to automate the human process of walking into the office, -updating a tree, compiling (and discovering the breakage), finding the -developer at fault, and complaining to them about the problem they had -introduced. With multiple platforms it was difficult for developers to -do the right thing (compile their potential change on all platforms); -the buildbot offered a way to help. - - <p>Another problem was when programmers would change the behavior of a -library without warning its users, or change internal aspects that -other code was (unfortunately) depending upon. Adding unit tests to -the codebase helps here: if an application's unit tests pass despite -changes in the libraries it uses, you can have more confidence that -the library changes haven't broken anything. Many developers -complained that the unit tests were inconvenient or took too long to -run: having the buildbot run them reduces the developer's workload to -a minimum. - - <p>In general, having more visibility into the project is always good, -and automation makes it easier for developers to do the right thing. -When everyone can see the status of the project, developers are -encouraged to keep the tree in good working order. Unit tests that -aren't run on a regular basis tend to suffer from bitrot just like -code does: exercising them on a regular basis helps to keep them -functioning and useful. - - <p>The current version of the Buildbot is additionally targeted at -distributed free-software projects, where resources and platforms are -only available when provided by interested volunteers. The buildslaves -are designed to require an absolute minimum of configuration, reducing -the effort a potential volunteer needs to expend to be able to -contribute a new test environment to the project. The goal is for -anyone who wishes that a given project would run on their favorite -platform should be able to offer that project a buildslave, running on -that platform, where they can verify that their portability code -works, and keeps working. - -<div class="node"> -<p><hr> -<a name="System-Architecture"></a> -Next: <a rel="next" accesskey="n" href="#Control-Flow">Control Flow</a>, -Previous: <a rel="previous" accesskey="p" href="#History-and-Philosophy">History and Philosophy</a>, -Up: <a rel="up" accesskey="u" href="#Introduction">Introduction</a> - -</div> - -<!-- node-name, next, previous, up --> -<h3 class="section">1.2 System Architecture</h3> - -<p>The Buildbot consists of a single <code>buildmaster</code> and one or more -<code>buildslaves</code>, connected in a star topology. The buildmaster -makes all decisions about what, when, and how to build. It sends -commands to be run on the build slaves, which simply execute the -commands and return the results. (certain steps involve more local -decision making, where the overhead of sending a lot of commands back -and forth would be inappropriate, but in general the buildmaster is -responsible for everything). - - <p>The buildmaster is usually fed <code>Changes</code> by some sort of version -control system (see <a href="#Change-Sources">Change Sources</a>), which may cause builds to be -run. As the builds are performed, various status messages are -produced, which are then sent to any registered Status Targets -(see <a href="#Status-Delivery">Status Delivery</a>). - -<!-- @image{FILENAME, WIDTH, HEIGHT, ALTTEXT, EXTENSION} --> - <div class="block-image"><img src="images/overview.png" alt="Overview Diagram"></div> - - <p>The buildmaster is configured and maintained by the “buildmaster -admin”, who is generally the project team member responsible for -build process issues. Each buildslave is maintained by a “buildslave -admin”, who do not need to be quite as involved. Generally slaves are -run by anyone who has an interest in seeing the project work well on -their favorite platform. - -<ul class="menu"> -<li><a accesskey="1" href="#BuildSlave-Connections">BuildSlave Connections</a> -<li><a accesskey="2" href="#Buildmaster-Architecture">Buildmaster Architecture</a> -<li><a accesskey="3" href="#Status-Delivery-Architecture">Status Delivery Architecture</a> -</ul> - -<div class="node"> -<p><hr> -<a name="BuildSlave-Connections"></a> -Next: <a rel="next" accesskey="n" href="#Buildmaster-Architecture">Buildmaster Architecture</a>, -Previous: <a rel="previous" accesskey="p" href="#System-Architecture">System Architecture</a>, -Up: <a rel="up" accesskey="u" href="#System-Architecture">System Architecture</a> - -</div> - -<h4 class="subsection">1.2.1 BuildSlave Connections</h4> - -<p>The buildslaves are typically run on a variety of separate machines, -at least one per platform of interest. These machines connect to the -buildmaster over a TCP connection to a publically-visible port. As a -result, the buildslaves can live behind a NAT box or similar -firewalls, as long as they can get to buildmaster. The TCP connections -are initiated by the buildslave and accepted by the buildmaster, but -commands and results travel both ways within this connection. The -buildmaster is always in charge, so all commands travel exclusively -from the buildmaster to the buildslave. - - <p>To perform builds, the buildslaves must typically obtain source code -from a CVS/SVN/etc repository. Therefore they must also be able to -reach the repository. The buildmaster provides instructions for -performing builds, but does not provide the source code itself. - - <div class="block-image"><img src="images/slaves.png" alt="BuildSlave Connections"></div> - -<div class="node"> -<p><hr> -<a name="Buildmaster-Architecture"></a> -Next: <a rel="next" accesskey="n" href="#Status-Delivery-Architecture">Status Delivery Architecture</a>, -Previous: <a rel="previous" accesskey="p" href="#BuildSlave-Connections">BuildSlave Connections</a>, -Up: <a rel="up" accesskey="u" href="#System-Architecture">System Architecture</a> - -</div> - -<h4 class="subsection">1.2.2 Buildmaster Architecture</h4> - -<p>The Buildmaster consists of several pieces: - - <div class="block-image"><img src="images/master.png" alt="BuildMaster Architecture"></div> - - <ul> -<li>Change Sources, which create a Change object each time something is -modified in the VC repository. Most ChangeSources listen for messages -from a hook script of some sort. Some sources actively poll the -repository on a regular basis. All Changes are fed to the Schedulers. - - <li>Schedulers, which decide when builds should be performed. They collect -Changes into BuildRequests, which are then queued for delivery to -Builders until a buildslave is available. - - <li>Builders, which control exactly <em>how</em> each build is performed -(with a series of BuildSteps, configured in a BuildFactory). Each -Build is run on a single buildslave. - - <li>Status plugins, which deliver information about the build results -through protocols like HTTP, mail, and IRC. - - </ul> - - <div class="block-image"><img src="images/slavebuilder.png" alt="SlaveBuilders"></div> - - <p>Each Builder is configured with a list of BuildSlaves that it will use -for its builds. These buildslaves are expected to behave identically: -the only reason to use multiple BuildSlaves for a single Builder is to -provide a measure of load-balancing. - - <p>Within a single BuildSlave, each Builder creates its own SlaveBuilder -instance. These SlaveBuilders operate independently from each other. -Each gets its own base directory to work in. It is quite common to -have many Builders sharing the same buildslave. For example, there -might be two buildslaves: one for i386, and a second for PowerPC. -There may then be a pair of Builders that do a full compile/test run, -one for each architecture, and a lone Builder that creates snapshot -source tarballs if the full builders complete successfully. The full -builders would each run on a single buildslave, whereas the tarball -creation step might run on either buildslave (since the platform -doesn't matter when creating source tarballs). In this case, the -mapping would look like: - -<pre class="example"> Builder(full-i386) -> BuildSlaves(slave-i386) - Builder(full-ppc) -> BuildSlaves(slave-ppc) - Builder(source-tarball) -> BuildSlaves(slave-i386, slave-ppc) -</pre> - <p>and each BuildSlave would have two SlaveBuilders inside it, one for a -full builder, and a second for the source-tarball builder. - - <p>Once a SlaveBuilder is available, the Builder pulls one or more -BuildRequests off its incoming queue. (It may pull more than one if it -determines that it can merge the requests together; for example, there -may be multiple requests to build the current HEAD revision). These -requests are merged into a single Build instance, which includes the -SourceStamp that describes what exact version of the source code -should be used for the build. The Build is then randomly assigned to a -free SlaveBuilder and the build begins. - - <p>The behaviour when BuildRequests are merged can be customized, see <a href="#Merging-BuildRequests">Merging BuildRequests</a>. - -<div class="node"> -<p><hr> -<a name="Status-Delivery-Architecture"></a> -Previous: <a rel="previous" accesskey="p" href="#Buildmaster-Architecture">Buildmaster Architecture</a>, -Up: <a rel="up" accesskey="u" href="#System-Architecture">System Architecture</a> - -</div> - -<h4 class="subsection">1.2.3 Status Delivery Architecture</h4> - -<p>The buildmaster maintains a central Status object, to which various -status plugins are connected. Through this Status object, a full -hierarchy of build status objects can be obtained. - - <div class="block-image"><img src="images/status.png" alt="Status Delivery"></div> - - <p>The configuration file controls which status plugins are active. Each -status plugin gets a reference to the top-level Status object. From -there they can request information on each Builder, Build, Step, and -LogFile. This query-on-demand interface is used by the html.Waterfall -plugin to create the main status page each time a web browser hits the -main URL. - - <p>The status plugins can also subscribe to hear about new Builds as they -occur: this is used by the MailNotifier to create new email messages -for each recently-completed Build. - - <p>The Status object records the status of old builds on disk in the -buildmaster's base directory. This allows it to return information -about historical builds. - - <p>There are also status objects that correspond to Schedulers and -BuildSlaves. These allow status plugins to report information about -upcoming builds, and the online/offline status of each buildslave. - -<div class="node"> -<p><hr> -<a name="Control-Flow"></a> -Previous: <a rel="previous" accesskey="p" href="#System-Architecture">System Architecture</a>, -Up: <a rel="up" accesskey="u" href="#Introduction">Introduction</a> - -</div> - -<!-- node-name, next, previous, up --> -<h3 class="section">1.3 Control Flow</h3> - -<p>A day in the life of the buildbot: - - <ul> -<li>A developer commits some source code changes to the repository. A hook -script or commit trigger of some sort sends information about this -change to the buildmaster through one of its configured Change -Sources. This notification might arrive via email, or over a network -connection (either initiated by the buildmaster as it “subscribes” -to changes, or by the commit trigger as it pushes Changes towards the -buildmaster). The Change contains information about who made the -change, what files were modified, which revision contains the change, -and any checkin comments. - - <li>The buildmaster distributes this change to all of its configured -Schedulers. Any “important” changes cause the “tree-stable-timer” -to be started, and the Change is added to a list of those that will go -into a new Build. When the timer expires, a Build is started on each -of a set of configured Builders, all compiling/testing the same source -code. Unless configured otherwise, all Builds run in parallel on the -various buildslaves. - - <li>The Build consists of a series of Steps. Each Step causes some number -of commands to be invoked on the remote buildslave associated with -that Builder. The first step is almost always to perform a checkout of -the appropriate revision from the same VC system that produced the -Change. The rest generally perform a compile and run unit tests. As -each Step runs, the buildslave reports back command output and return -status to the buildmaster. - - <li>As the Build runs, status messages like “Build Started”, “Step -Started”, “Build Finished”, etc, are published to a collection of -Status Targets. One of these targets is usually the HTML “Waterfall” -display, which shows a chronological list of events, and summarizes -the results of the most recent build at the top of each column. -Developers can periodically check this page to see how their changes -have fared. If they see red, they know that they've made a mistake and -need to fix it. If they see green, they know that they've done their -duty and don't need to worry about their change breaking anything. - - <li>If a MailNotifier status target is active, the completion of a build -will cause email to be sent to any developers whose Changes were -incorporated into this Build. The MailNotifier can be configured to -only send mail upon failing builds, or for builds which have just -transitioned from passing to failing. Other status targets can provide -similar real-time notification via different communication channels, -like IRC. - - </ul> - -<div class="node"> -<p><hr> -<a name="Installation"></a> -Next: <a rel="next" accesskey="n" href="#Concepts">Concepts</a>, -Previous: <a rel="previous" accesskey="p" href="#Introduction">Introduction</a>, -Up: <a rel="up" accesskey="u" href="#Top">Top</a> - -</div> - -<h2 class="chapter">2 Installation</h2> - -<ul class="menu"> -<li><a accesskey="1" href="#Requirements">Requirements</a> -<li><a accesskey="2" href="#Installing-the-code">Installing the code</a> -<li><a accesskey="3" href="#Creating-a-buildmaster">Creating a buildmaster</a> -<li><a accesskey="4" href="#Upgrading-an-Existing-Buildmaster">Upgrading an Existing Buildmaster</a> -<li><a accesskey="5" href="#Creating-a-buildslave">Creating a buildslave</a> -<li><a accesskey="6" href="#Launching-the-daemons">Launching the daemons</a> -<li><a accesskey="7" href="#Logfiles">Logfiles</a> -<li><a accesskey="8" href="#Shutdown">Shutdown</a> -<li><a accesskey="9" href="#Maintenance">Maintenance</a> -<li><a href="#Troubleshooting">Troubleshooting</a> -</ul> - -<div class="node"> -<p><hr> -<a name="Requirements"></a> -Next: <a rel="next" accesskey="n" href="#Installing-the-code">Installing the code</a>, -Previous: <a rel="previous" accesskey="p" href="#Installation">Installation</a>, -Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> - -</div> - -<h3 class="section">2.1 Requirements</h3> - -<p>At a bare minimum, you'll need the following (for both the buildmaster -and a buildslave): - - <ul> -<li>Python: http://www.python.org - - <p>Buildbot requires python-2.3 or later, and is primarily developed -against python-2.4. It is also tested against python-2.5 . - - <li>Twisted: http://twistedmatrix.com - - <p>Both the buildmaster and the buildslaves require Twisted-2.0.x or -later. It has been tested against all releases of Twisted up to -Twisted-2.5.0 (the most recent as of this writing). As always, the -most recent version is recommended. - - <p>Twisted is delivered as a collection of subpackages. You'll need at -least "Twisted" (the core package), and you'll also want TwistedMail, -TwistedWeb, and TwistedWords (for sending email, serving a web status -page, and delivering build status via IRC, respectively). You might -also want TwistedConch (for the encrypted Manhole debug port). Note -that Twisted requires ZopeInterface to be installed as well. - - </ul> - - <p>Certain other packages may be useful on the system running the -buildmaster: - - <ul> -<li>CVSToys: http://purl.net/net/CVSToys - - <p>If your buildmaster uses FreshCVSSource to receive change notification -from a cvstoys daemon, it will require CVSToys be installed (tested -with CVSToys-1.0.10). If the it doesn't use that source (i.e. if you -only use a mail-parsing change source, or the SVN notification -script), you will not need CVSToys. - - </ul> - - <p>And of course, your project's build process will impose additional -requirements on the buildslaves. These hosts must have all the tools -necessary to compile and test your project's source code. - -<div class="node"> -<p><hr> -<a name="Installing-the-code"></a> -Next: <a rel="next" accesskey="n" href="#Creating-a-buildmaster">Creating a buildmaster</a>, -Previous: <a rel="previous" accesskey="p" href="#Requirements">Requirements</a>, -Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> - -</div> - -<h3 class="section">2.2 Installing the code</h3> - -<p><a name="index-installation-3"></a> -The Buildbot is installed using the standard python <code>distutils</code> -module. After unpacking the tarball, the process is: - -<pre class="example"> python setup.py build - python setup.py install -</pre> - <p>where the install step may need to be done as root. This will put the -bulk of the code in somewhere like -/usr/lib/python2.3/site-packages/buildbot . It will also install the -<code>buildbot</code> command-line tool in /usr/bin/buildbot. - - <p>To test this, shift to a different directory (like /tmp), and run: - -<pre class="example"> buildbot --version -</pre> - <p>If it shows you the versions of Buildbot and Twisted, the install went -ok. If it says <code>no such command</code> or it gets an <code>ImportError</code> -when it tries to load the libaries, then something went wrong. -<code>pydoc buildbot</code> is another useful diagnostic tool. - - <p>Windows users will find these files in other places. You will need to -make sure that python can find the libraries, and will probably find -it convenient to have <code>buildbot</code> on your PATH. - - <p>If you wish, you can run the buildbot unit test suite like this: - -<pre class="example"> PYTHONPATH=. trial buildbot.test -</pre> - <p>This should run up to 192 tests, depending upon what VC tools you have -installed. On my desktop machine it takes about five minutes to -complete. Nothing should fail, a few might be skipped. If any of the -tests fail, you should stop and investigate the cause before -continuing the installation process, as it will probably be easier to -track down the bug early. - - <p>If you cannot or do not wish to install the buildbot into a site-wide -location like <samp><span class="file">/usr</span></samp> or <samp><span class="file">/usr/local</span></samp>, you can also install -it into the account's home directory. Do the install command like -this: - -<pre class="example"> python setup.py install --home=~ -</pre> - <p>That will populate <samp><span class="file">~/lib/python</span></samp> and create -<samp><span class="file">~/bin/buildbot</span></samp>. Make sure this lib directory is on your -<code>PYTHONPATH</code>. - -<div class="node"> -<p><hr> -<a name="Creating-a-buildmaster"></a> -Next: <a rel="next" accesskey="n" href="#Upgrading-an-Existing-Buildmaster">Upgrading an Existing Buildmaster</a>, -Previous: <a rel="previous" accesskey="p" href="#Installing-the-code">Installing the code</a>, -Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> - -</div> - -<h3 class="section">2.3 Creating a buildmaster</h3> - -<p>As you learned earlier (see <a href="#System-Architecture">System Architecture</a>), the buildmaster -runs on a central host (usually one that is publically visible, so -everybody can check on the status of the project), and controls all -aspects of the buildbot system. Let us call this host -<code>buildbot.example.org</code>. - - <p>You may wish to create a separate user account for the buildmaster, -perhaps named <code>buildmaster</code>. This can help keep your personal -configuration distinct from that of the buildmaster and is useful if -you have to use a mail-based notification system (see <a href="#Change-Sources">Change Sources</a>). However, the Buildbot will work just fine with your regular -user account. - - <p>You need to choose a directory for the buildmaster, called the -<code>basedir</code>. This directory will be owned by the buildmaster, which -will use configuration files therein, and create status files as it -runs. <samp><span class="file">~/Buildbot</span></samp> is a likely value. If you run multiple -buildmasters in the same account, or if you run both masters and -slaves, you may want a more distinctive name like -<samp><span class="file">~/Buildbot/master/gnomovision</span></samp> or -<samp><span class="file">~/Buildmasters/fooproject</span></samp>. If you are using a separate user -account, this might just be <samp><span class="file">~buildmaster/masters/fooproject</span></samp>. - - <p>Once you've picked a directory, use the <samp><span class="command">buildbot -create-master</span></samp> command to create the directory and populate it with -startup files: - -<pre class="example"> buildbot create-master <var>basedir</var> -</pre> - <p>You will need to create a configuration file (see <a href="#Configuration">Configuration</a>) -before starting the buildmaster. Most of the rest of this manual is -dedicated to explaining how to do this. A sample configuration file is -placed in the working directory, named <samp><span class="file">master.cfg.sample</span></samp>, which -can be copied to <samp><span class="file">master.cfg</span></samp> and edited to suit your purposes. - - <p>(Internal details: This command creates a file named -<samp><span class="file">buildbot.tac</span></samp> that contains all the state necessary to create -the buildmaster. Twisted has a tool called <code>twistd</code> which can use -this .tac file to create and launch a buildmaster instance. twistd -takes care of logging and daemonization (running the program in the -background). <samp><span class="file">/usr/bin/buildbot</span></samp> is a front end which runs twistd -for you.) - - <p>In addition to <samp><span class="file">buildbot.tac</span></samp>, a small <samp><span class="file">Makefile.sample</span></samp> is -installed. This can be used as the basis for customized daemon startup, -See <a href="#Launching-the-daemons">Launching the daemons</a>. - -<div class="node"> -<p><hr> -<a name="Upgrading-an-Existing-Buildmaster"></a> -Next: <a rel="next" accesskey="n" href="#Creating-a-buildslave">Creating a buildslave</a>, -Previous: <a rel="previous" accesskey="p" href="#Creating-a-buildmaster">Creating a buildmaster</a>, -Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> - -</div> - -<h3 class="section">2.4 Upgrading an Existing Buildmaster</h3> - -<p>If you have just installed a new version of the Buildbot code, and you -have buildmasters that were created using an older version, you'll -need to upgrade these buildmasters before you can use them. The -upgrade process adds and modifies files in the buildmaster's base -directory to make it compatible with the new code. - -<pre class="example"> buildbot upgrade-master <var>basedir</var> -</pre> - <p>This command will also scan your <samp><span class="file">master.cfg</span></samp> file for -incompatbilities (by loading it and printing any errors or deprecation -warnings that occur). Each buildbot release tries to be compatible -with configurations that worked cleanly (i.e. without deprecation -warnings) on the previous release: any functions or classes that are -to be removed will first be deprecated in a release, to give users a -chance to start using their replacement. - - <p>The 0.7.6 release introduced the <samp><span class="file">public_html/</span></samp> directory, which -contains <samp><span class="file">index.html</span></samp> and other files served by the -<code>WebStatus</code> and <code>Waterfall</code> status displays. The -<code>upgrade-master</code> command will create these files if they do not -already exist. It will not modify existing copies, but it will write a -new copy in e.g. <samp><span class="file">index.html.new</span></samp> if the new version differs from -the version that already exists. - - <p>The <code>upgrade-master</code> command is idempotent. It is safe to run it -multiple times. After each upgrade of the buildbot code, you should -use <code>upgrade-master</code> on all your buildmasters. - -<div class="node"> -<p><hr> -<a name="Creating-a-buildslave"></a> -Next: <a rel="next" accesskey="n" href="#Launching-the-daemons">Launching the daemons</a>, -Previous: <a rel="previous" accesskey="p" href="#Upgrading-an-Existing-Buildmaster">Upgrading an Existing Buildmaster</a>, -Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> - -</div> - -<h3 class="section">2.5 Creating a buildslave</h3> - -<p>Typically, you will be adding a buildslave to an existing buildmaster, -to provide additional architecture coverage. The buildbot -administrator will give you several pieces of information necessary to -connect to the buildmaster. You should also be somewhat familiar with -the project being tested, so you can troubleshoot build problems -locally. - - <p>The buildbot exists to make sure that the project's stated “how to -build it” process actually works. To this end, the buildslave should -run in an environment just like that of your regular developers. -Typically the project build process is documented somewhere -(<samp><span class="file">README</span></samp>, <samp><span class="file">INSTALL</span></samp>, etc), in a document that should -mention all library dependencies and contain a basic set of build -instructions. This document will be useful as you configure the host -and account in which the buildslave runs. - - <p>Here's a good checklist for setting up a buildslave: - - <ol type=1 start=1> -<li>Set up the account - - <p>It is recommended (although not mandatory) to set up a separate user -account for the buildslave. This account is frequently named -<code>buildbot</code> or <code>buildslave</code>. This serves to isolate your -personal working environment from that of the slave's, and helps to -minimize the security threat posed by letting possibly-unknown -contributors run arbitrary code on your system. The account should -have a minimum of fancy init scripts. - - <li>Install the buildbot code - - <p>Follow the instructions given earlier (see <a href="#Installing-the-code">Installing the code</a>). -If you use a separate buildslave account, and you didn't install the -buildbot code to a shared location, then you will need to install it -with <code>--home=~</code> for each account that needs it. - - <li>Set up the host - - <p>Make sure the host can actually reach the buildmaster. Usually the -buildmaster is running a status webserver on the same machine, so -simply point your web browser at it and see if you can get there. -Install whatever additional packages or libraries the project's -INSTALL document advises. (or not: if your buildslave is supposed to -make sure that building without optional libraries still works, then -don't install those libraries). - - <p>Again, these libraries don't necessarily have to be installed to a -site-wide shared location, but they must be available to your build -process. Accomplishing this is usually very specific to the build -process, so installing them to <samp><span class="file">/usr</span></samp> or <samp><span class="file">/usr/local</span></samp> is -usually the best approach. - - <li>Test the build process - - <p>Follow the instructions in the INSTALL document, in the buildslave's -account. Perform a full CVS (or whatever) checkout, configure, make, -run tests, etc. Confirm that the build works without manual fussing. -If it doesn't work when you do it by hand, it will be unlikely to work -when the buildbot attempts to do it in an automated fashion. - - <li>Choose a base directory - - <p>This should be somewhere in the buildslave's account, typically named -after the project which is being tested. The buildslave will not touch -any file outside of this directory. Something like <samp><span class="file">~/Buildbot</span></samp> -or <samp><span class="file">~/Buildslaves/fooproject</span></samp> is appropriate. - - <li>Get the buildmaster host/port, botname, and password - - <p>When the buildbot admin configures the buildmaster to accept and use -your buildslave, they will provide you with the following pieces of -information: - - <ul> -<li>your buildslave's name -<li>the password assigned to your buildslave -<li>the hostname and port number of the buildmaster, i.e. buildbot.example.org:8007 -</ul> - - <li>Create the buildslave - - <p>Now run the 'buildbot' command as follows: - - <pre class="example"> buildbot create-slave <var>BASEDIR</var> <var>MASTERHOST</var>:<var>PORT</var> <var>SLAVENAME</var> <var>PASSWORD</var> -</pre> - <p>This will create the base directory and a collection of files inside, -including the <samp><span class="file">buildbot.tac</span></samp> file that contains all the -information you passed to the <code>buildbot</code> command. - - <li>Fill in the hostinfo files - - <p>When it first connects, the buildslave will send a few files up to the -buildmaster which describe the host that it is running on. These files -are presented on the web status display so that developers have more -information to reproduce any test failures that are witnessed by the -buildbot. There are sample files in the <samp><span class="file">info</span></samp> subdirectory of -the buildbot's base directory. You should edit these to correctly -describe you and your host. - - <p><samp><span class="file">BASEDIR/info/admin</span></samp> should contain your name and email address. -This is the “buildslave admin address”, and will be visible from the -build status page (so you may wish to munge it a bit if -address-harvesting spambots are a concern). - - <p><samp><span class="file">BASEDIR/info/host</span></samp> should be filled with a brief description of -the host: OS, version, memory size, CPU speed, versions of relevant -libraries installed, and finally the version of the buildbot code -which is running the buildslave. - - <p>If you run many buildslaves, you may want to create a single -<samp><span class="file">~buildslave/info</span></samp> file and share it among all the buildslaves -with symlinks. - - </ol> - -<ul class="menu"> -<li><a accesskey="1" href="#Buildslave-Options">Buildslave Options</a> -</ul> - -<div class="node"> -<p><hr> -<a name="Buildslave-Options"></a> -Previous: <a rel="previous" accesskey="p" href="#Creating-a-buildslave">Creating a buildslave</a>, -Up: <a rel="up" accesskey="u" href="#Creating-a-buildslave">Creating a buildslave</a> - -</div> - -<h4 class="subsection">2.5.1 Buildslave Options</h4> - -<p>There are a handful of options you might want to use when creating the -buildslave with the <samp><span class="command">buildbot create-slave <options> DIR <params></span></samp> -command. You can type <samp><span class="command">buildbot create-slave --help</span></samp> for a summary. -To use these, just include them on the <samp><span class="command">buildbot create-slave</span></samp> -command line, like this: - -<pre class="example"> buildbot create-slave --umask=022 ~/buildslave buildmaster.example.org:42012 myslavename mypasswd -</pre> - <dl> -<dt><code>--usepty</code><dd>This is a boolean flag that tells the buildslave whether to launch child -processes in a PTY or with regular pipes (the default) when the master does not -specify. This option is deprecated, as this particular parameter is better -specified on the master. - - <br><dt><code>--umask</code><dd>This is a string (generally an octal representation of an integer) -which will cause the buildslave process' “umask” value to be set -shortly after initialization. The “twistd” daemonization utility -forces the umask to 077 at startup (which means that all files created -by the buildslave or its child processes will be unreadable by any -user other than the buildslave account). If you want build products to -be readable by other accounts, you can add <code>--umask=022</code> to tell -the buildslave to fix the umask after twistd clobbers it. If you want -build products to be <em>writable</em> by other accounts too, use -<code>--umask=000</code>, but this is likely to be a security problem. - - <br><dt><code>--keepalive</code><dd>This is a number that indicates how frequently “keepalive” messages -should be sent from the buildslave to the buildmaster, expressed in -seconds. The default (600) causes a message to be sent to the -buildmaster at least once every 10 minutes. To set this to a lower -value, use e.g. <code>--keepalive=120</code>. - - <p>If the buildslave is behind a NAT box or stateful firewall, these -messages may help to keep the connection alive: some NAT boxes tend to -forget about a connection if it has not been used in a while. When -this happens, the buildmaster will think that the buildslave has -disappeared, and builds will time out. Meanwhile the buildslave will -not realize than anything is wrong. - - <br><dt><code>--maxdelay</code><dd>This is a number that indicates the maximum amount of time the -buildslave will wait between connection attempts, expressed in -seconds. The default (300) causes the buildslave to wait at most 5 -minutes before trying to connect to the buildmaster again. - - <br><dt><code>--log-size</code><dd>This is the size in bytes when to rotate the Twisted log files. - - <br><dt><code>--log-count</code><dd>This is the number of log rotations to keep around. You can either -specify a number or <code>None</code> (the default) to keep all -<samp><span class="file">twistd.log</span></samp> files around. - - </dl> - -<div class="node"> -<p><hr> -<a name="Launching-the-daemons"></a> -Next: <a rel="next" accesskey="n" href="#Logfiles">Logfiles</a>, -Previous: <a rel="previous" accesskey="p" href="#Creating-a-buildslave">Creating a buildslave</a>, -Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> - -</div> - -<h3 class="section">2.6 Launching the daemons</h3> - -<p>Both the buildmaster and the buildslave run as daemon programs. To -launch them, pass the working directory to the <code>buildbot</code> -command: - -<pre class="example"> buildbot start <var>BASEDIR</var> -</pre> - <p>This command will start the daemon and then return, so normally it -will not produce any output. To verify that the programs are indeed -running, look for a pair of files named <samp><span class="file">twistd.log</span></samp> and -<samp><span class="file">twistd.pid</span></samp> that should be created in the working directory. -<samp><span class="file">twistd.pid</span></samp> contains the process ID of the newly-spawned daemon. - - <p>When the buildslave connects to the buildmaster, new directories will -start appearing in its base directory. The buildmaster tells the slave -to create a directory for each Builder which will be using that slave. -All build operations are performed within these directories: CVS -checkouts, compiles, and tests. - - <p>Once you get everything running, you will want to arrange for the -buildbot daemons to be started at boot time. One way is to use -<code>cron</code>, by putting them in a @reboot crontab entry<a rel="footnote" href="#fn-1" name="fnd-1"><sup>1</sup></a>: - -<pre class="example"> @reboot buildbot start <var>BASEDIR</var> -</pre> - <p>When you run <samp><span class="command">crontab</span></samp> to set this up, remember to do it as -the buildmaster or buildslave account! If you add this to your crontab -when running as your regular account (or worse yet, root), then the -daemon will run as the wrong user, quite possibly as one with more -authority than you intended to provide. - - <p>It is important to remember that the environment provided to cron jobs -and init scripts can be quite different that your normal runtime. -There may be fewer environment variables specified, and the PATH may -be shorter than usual. It is a good idea to test out this method of -launching the buildslave by using a cron job with a time in the near -future, with the same command, and then check <samp><span class="file">twistd.log</span></samp> to -make sure the slave actually started correctly. Common problems here -are for <samp><span class="file">/usr/local</span></samp> or <samp><span class="file">~/bin</span></samp> to not be on your -<code>PATH</code>, or for <code>PYTHONPATH</code> to not be set correctly. -Sometimes <code>HOME</code> is messed up too. - - <p>To modify the way the daemons are started (perhaps you want to set -some environment variables first, or perform some cleanup each time), -you can create a file named <samp><span class="file">Makefile.buildbot</span></samp> in the base -directory. When the <samp><span class="file">buildbot</span></samp> front-end tool is told to -<samp><span class="command">start</span></samp> the daemon, and it sees this file (and -<samp><span class="file">/usr/bin/make</span></samp> exists), it will do <samp><span class="command">make -f -Makefile.buildbot start</span></samp> instead of its usual action (which involves -running <samp><span class="command">twistd</span></samp>). When the buildmaster or buildslave is -installed, a <samp><span class="file">Makefile.sample</span></samp> is created which implements the -same behavior as the the <samp><span class="file">buildbot</span></samp> tool uses, so if you want to -customize the process, just copy <samp><span class="file">Makefile.sample</span></samp> to -<samp><span class="file">Makefile.buildbot</span></samp> and edit it as necessary. - - <p>Some distributions may include conveniences to make starting buildbot -at boot time easy. For instance, with the default buildbot package in -Debian-based distributions, you may only need to modify -<code>/etc/default/buildbot</code> (see also <code>/etc/init.d/buildbot</code>, which -reads the configuration in <code>/etc/default/buildbot</code>). - -<div class="node"> -<p><hr> -<a name="Logfiles"></a> -Next: <a rel="next" accesskey="n" href="#Shutdown">Shutdown</a>, -Previous: <a rel="previous" accesskey="p" href="#Launching-the-daemons">Launching the daemons</a>, -Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> - -</div> - -<h3 class="section">2.7 Logfiles</h3> - -<p><a name="index-logfiles-4"></a> -While a buildbot daemon runs, it emits text to a logfile, named -<samp><span class="file">twistd.log</span></samp>. A command like <code>tail -f twistd.log</code> is useful -to watch the command output as it runs. - - <p>The buildmaster will announce any errors with its configuration file -in the logfile, so it is a good idea to look at the log at startup -time to check for any problems. Most buildmaster activities will cause -lines to be added to the log. - -<div class="node"> -<p><hr> -<a name="Shutdown"></a> -Next: <a rel="next" accesskey="n" href="#Maintenance">Maintenance</a>, -Previous: <a rel="previous" accesskey="p" href="#Logfiles">Logfiles</a>, -Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> - -</div> - -<h3 class="section">2.8 Shutdown</h3> - -<p>To stop a buildmaster or buildslave manually, use: - -<pre class="example"> buildbot stop <var>BASEDIR</var> -</pre> - <p>This simply looks for the <samp><span class="file">twistd.pid</span></samp> file and kills whatever -process is identified within. - - <p>At system shutdown, all processes are sent a <code>SIGKILL</code>. The -buildmaster and buildslave will respond to this by shutting down -normally. - - <p>The buildmaster will respond to a <code>SIGHUP</code> by re-reading its -config file. Of course, this only works on unix-like systems with -signal support, and won't work on Windows. The following shortcut is -available: - -<pre class="example"> buildbot reconfig <var>BASEDIR</var> -</pre> - <p>When you update the Buildbot code to a new release, you will need to -restart the buildmaster and/or buildslave before it can take advantage -of the new code. You can do a <code>buildbot stop </code><var>BASEDIR</var> and -<code>buildbot start </code><var>BASEDIR</var> in quick succession, or you can -use the <code>restart</code> shortcut, which does both steps for you: - -<pre class="example"> buildbot restart <var>BASEDIR</var> -</pre> - <p>There are certain configuration changes that are not handled cleanly -by <code>buildbot reconfig</code>. If this occurs, <code>buildbot restart</code> -is a more robust tool to fully switch over to the new configuration. - - <p><code>buildbot restart</code> may also be used to start a stopped Buildbot -instance. This behaviour is useful when writing scripts that stop, start -and restart Buildbot. - - <p>A buildslave may also be gracefully shutdown from the -see <a href="#WebStatus">WebStatus</a> status plugin. This is useful to shutdown a -buildslave without interrupting any current builds. The buildmaster -will wait until the buildslave is finished all its current builds, and -will then tell the buildslave to shutdown. - -<div class="node"> -<p><hr> -<a name="Maintenance"></a> -Next: <a rel="next" accesskey="n" href="#Troubleshooting">Troubleshooting</a>, -Previous: <a rel="previous" accesskey="p" href="#Shutdown">Shutdown</a>, -Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> - -</div> - -<h3 class="section">2.9 Maintenance</h3> - -<p>It is a good idea to check the buildmaster's status page every once in -a while, to see if your buildslave is still online. Eventually the -buildbot will probably be enhanced to send you email (via the -<samp><span class="file">info/admin</span></samp> email address) when the slave has been offline for -more than a few hours. - - <p>If you find you can no longer provide a buildslave to the project, please -let the project admins know, so they can put out a call for a -replacement. - - <p>The Buildbot records status and logs output continually, each time a -build is performed. The status tends to be small, but the build logs -can become quite large. Each build and log are recorded in a separate -file, arranged hierarchically under the buildmaster's base directory. -To prevent these files from growing without bound, you should -periodically delete old build logs. A simple cron job to delete -anything older than, say, two weeks should do the job. The only trick -is to leave the <samp><span class="file">buildbot.tac</span></samp> and other support files alone, for -which find's <code>-mindepth</code> argument helps skip everything in the -top directory. You can use something like the following: - -<pre class="example"> @weekly cd BASEDIR && find . -mindepth 2 i-path './public_html/*' -prune -o -type f -mtime +14 -exec rm {} \; - @weekly cd BASEDIR && find twistd.log* -mtime +14 -exec rm {} \; -</pre> - <div class="node"> -<p><hr> -<a name="Troubleshooting"></a> -Previous: <a rel="previous" accesskey="p" href="#Maintenance">Maintenance</a>, -Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> - -</div> - -<h3 class="section">2.10 Troubleshooting</h3> - -<p>Here are a few hints on diagnosing common problems. - -<ul class="menu"> -<li><a accesskey="1" href="#Starting-the-buildslave">Starting the buildslave</a> -<li><a accesskey="2" href="#Connecting-to-the-buildmaster">Connecting to the buildmaster</a> -<li><a accesskey="3" href="#Forcing-Builds">Forcing Builds</a> -</ul> - -<div class="node"> -<p><hr> -<a name="Starting-the-buildslave"></a> -Next: <a rel="next" accesskey="n" href="#Connecting-to-the-buildmaster">Connecting to the buildmaster</a>, -Previous: <a rel="previous" accesskey="p" href="#Troubleshooting">Troubleshooting</a>, -Up: <a rel="up" accesskey="u" href="#Troubleshooting">Troubleshooting</a> - -</div> - -<h4 class="subsection">2.10.1 Starting the buildslave</h4> - -<p>Cron jobs are typically run with a minimal shell (<samp><span class="file">/bin/sh</span></samp>, not -<samp><span class="file">/bin/bash</span></samp>), and tilde expansion is not always performed in such -commands. You may want to use explicit paths, because the <code>PATH</code> -is usually quite short and doesn't include anything set by your -shell's startup scripts (<samp><span class="file">.profile</span></samp>, <samp><span class="file">.bashrc</span></samp>, etc). If -you've installed buildbot (or other python libraries) to an unusual -location, you may need to add a <code>PYTHONPATH</code> specification (note -that python will do tilde-expansion on <code>PYTHONPATH</code> elements by -itself). Sometimes it is safer to fully-specify everything: - -<pre class="example"> @reboot PYTHONPATH=~/lib/python /usr/local/bin/buildbot start /usr/home/buildbot/basedir -</pre> - <p>Take the time to get the @reboot job set up. Otherwise, things will work -fine for a while, but the first power outage or system reboot you have will -stop the buildslave with nothing but the cries of sorrowful developers to -remind you that it has gone away. - -<div class="node"> -<p><hr> -<a name="Connecting-to-the-buildmaster"></a> -Next: <a rel="next" accesskey="n" href="#Forcing-Builds">Forcing Builds</a>, -Previous: <a rel="previous" accesskey="p" href="#Starting-the-buildslave">Starting the buildslave</a>, -Up: <a rel="up" accesskey="u" href="#Troubleshooting">Troubleshooting</a> - -</div> - -<h4 class="subsection">2.10.2 Connecting to the buildmaster</h4> - -<p>If the buildslave cannot connect to the buildmaster, the reason should -be described in the <samp><span class="file">twistd.log</span></samp> logfile. Some common problems -are an incorrect master hostname or port number, or a mistyped bot -name or password. If the buildslave loses the connection to the -master, it is supposed to attempt to reconnect with an -exponentially-increasing backoff. Each attempt (and the time of the -next attempt) will be logged. If you get impatient, just manually stop -and re-start the buildslave. - - <p>When the buildmaster is restarted, all slaves will be disconnected, -and will attempt to reconnect as usual. The reconnect time will depend -upon how long the buildmaster is offline (i.e. how far up the -exponential backoff curve the slaves have travelled). Again, -<code>buildbot stop </code><var>BASEDIR</var><code>; buildbot start </code><var>BASEDIR</var> will -speed up the process. - -<div class="node"> -<p><hr> -<a name="Forcing-Builds"></a> -Previous: <a rel="previous" accesskey="p" href="#Connecting-to-the-buildmaster">Connecting to the buildmaster</a>, -Up: <a rel="up" accesskey="u" href="#Troubleshooting">Troubleshooting</a> - -</div> - -<h4 class="subsection">2.10.3 Forcing Builds</h4> - -<p>From the buildmaster's main status web page, you can force a build to -be run on your build slave. Figure out which column is for a builder -that runs on your slave, click on that builder's name, and the page -that comes up will have a “Force Build” button. Fill in the form, -hit the button, and a moment later you should see your slave's -<samp><span class="file">twistd.log</span></samp> filling with commands being run. Using <code>pstree</code> -or <code>top</code> should also reveal the cvs/make/gcc/etc processes being -run by the buildslave. Note that the same web page should also show -the <samp><span class="file">admin</span></samp> and <samp><span class="file">host</span></samp> information files that you configured -earlier. - -<div class="node"> -<p><hr> -<a name="Concepts"></a> -Next: <a rel="next" accesskey="n" href="#Configuration">Configuration</a>, -Previous: <a rel="previous" accesskey="p" href="#Installation">Installation</a>, -Up: <a rel="up" accesskey="u" href="#Top">Top</a> - -</div> - -<h2 class="chapter">3 Concepts</h2> - -<p>This chapter defines some of the basic concepts that the Buildbot -uses. You'll need to understand how the Buildbot sees the world to -configure it properly. - -<ul class="menu"> -<li><a accesskey="1" href="#Version-Control-Systems">Version Control Systems</a> -<li><a accesskey="2" href="#Schedulers">Schedulers</a> -<li><a accesskey="3" href="#BuildSet">BuildSet</a> -<li><a accesskey="4" href="#BuildRequest">BuildRequest</a> -<li><a accesskey="5" href="#Builder">Builder</a> -<li><a accesskey="6" href="#Users">Users</a> -<li><a accesskey="7" href="#Build-Properties">Build Properties</a> -</ul> - -<div class="node"> -<p><hr> -<a name="Version-Control-Systems"></a> -Next: <a rel="next" accesskey="n" href="#Schedulers">Schedulers</a>, -Previous: <a rel="previous" accesskey="p" href="#Concepts">Concepts</a>, -Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a> - -</div> - -<h3 class="section">3.1 Version Control Systems</h3> - -<p><a name="index-Version-Control-5"></a> -These source trees come from a Version Control System of some kind. -CVS and Subversion are two popular ones, but the Buildbot supports -others. All VC systems have some notion of an upstream -<code>repository</code> which acts as a server<a rel="footnote" href="#fn-2" name="fnd-2"><sup>2</sup></a>, from which clients -can obtain source trees according to various parameters. The VC -repository provides source trees of various projects, for different -branches, and from various points in time. The first thing we have to -do is to specify which source tree we want to get. - -<ul class="menu"> -<li><a accesskey="1" href="#Generalizing-VC-Systems">Generalizing VC Systems</a> -<li><a accesskey="2" href="#Source-Tree-Specifications">Source Tree Specifications</a> -<li><a accesskey="3" href="#How-Different-VC-Systems-Specify-Sources">How Different VC Systems Specify Sources</a> -<li><a accesskey="4" href="#Attributes-of-Changes">Attributes of Changes</a> -</ul> - -<div class="node"> -<p><hr> -<a name="Generalizing-VC-Systems"></a> -Next: <a rel="next" accesskey="n" href="#Source-Tree-Specifications">Source Tree Specifications</a>, -Previous: <a rel="previous" accesskey="p" href="#Version-Control-Systems">Version Control Systems</a>, -Up: <a rel="up" accesskey="u" href="#Version-Control-Systems">Version Control Systems</a> - -</div> - -<h4 class="subsection">3.1.1 Generalizing VC Systems</h4> - -<p>For the purposes of the Buildbot, we will try to generalize all VC -systems as having repositories that each provide sources for a variety -of projects. Each project is defined as a directory tree with source -files. The individual files may each have revisions, but we ignore -that and treat the project as a whole as having a set of revisions -(CVS is really the only VC system still in widespread use that has -per-file revisions.. everything modern has moved to atomic tree-wide -changesets). Each time someone commits a change to the project, a new -revision becomes available. These revisions can be described by a -tuple with two items: the first is a branch tag, and the second is -some kind of revision stamp or timestamp. Complex projects may have -multiple branch tags, but there is always a default branch. The -timestamp may be an actual timestamp (such as the -D option to CVS), -or it may be a monotonically-increasing transaction number (such as -the change number used by SVN and P4, or the revision number used by -Arch/Baz/Bazaar, or a labeled tag used in CVS)<a rel="footnote" href="#fn-3" name="fnd-3"><sup>3</sup></a>. The SHA1 revision ID used by Monotone, Mercurial, and Git is -also a kind of revision stamp, in that it specifies a unique copy of -the source tree, as does a Darcs “context” file. - - <p>When we aren't intending to make any changes to the sources we check out -(at least not any that need to be committed back upstream), there are two -basic ways to use a VC system: - - <ul> -<li>Retrieve a specific set of source revisions: some tag or key is used -to index this set, which is fixed and cannot be changed by subsequent -developers committing new changes to the tree. Releases are built from -tagged revisions like this, so that they can be rebuilt again later -(probably with controlled modifications). -<li>Retrieve the latest sources along a specific branch: some tag is used -to indicate which branch is to be used, but within that constraint we want -to get the latest revisions. -</ul> - - <p>Build personnel or CM staff typically use the first approach: the -build that results is (ideally) completely specified by the two -parameters given to the VC system: repository and revision tag. This -gives QA and end-users something concrete to point at when reporting -bugs. Release engineers are also reportedly fond of shipping code that -can be traced back to a concise revision tag of some sort. - - <p>Developers are more likely to use the second approach: each morning -the developer does an update to pull in the changes committed by the -team over the last day. These builds are not easy to fully specify: it -depends upon exactly when you did a checkout, and upon what local -changes the developer has in their tree. Developers do not normally -tag each build they produce, because there is usually significant -overhead involved in creating these tags. Recreating the trees used by -one of these builds can be a challenge. Some VC systems may provide -implicit tags (like a revision number), while others may allow the use -of timestamps to mean “the state of the tree at time X” as opposed -to a tree-state that has been explicitly marked. - - <p>The Buildbot is designed to help developers, so it usually works in -terms of <em>the latest</em> sources as opposed to specific tagged -revisions. However, it would really prefer to build from reproducible -source trees, so implicit revisions are used whenever possible. - -<div class="node"> -<p><hr> -<a name="Source-Tree-Specifications"></a> -Next: <a rel="next" accesskey="n" href="#How-Different-VC-Systems-Specify-Sources">How Different VC Systems Specify Sources</a>, -Previous: <a rel="previous" accesskey="p" href="#Generalizing-VC-Systems">Generalizing VC Systems</a>, -Up: <a rel="up" accesskey="u" href="#Version-Control-Systems">Version Control Systems</a> - -</div> - -<h4 class="subsection">3.1.2 Source Tree Specifications</h4> - -<p>So for the Buildbot's purposes we treat each VC system as a server -which can take a list of specifications as input and produce a source -tree as output. Some of these specifications are static: they are -attributes of the builder and do not change over time. Others are more -variable: each build will have a different value. The repository is -changed over time by a sequence of Changes, each of which represents a -single developer making changes to some set of files. These Changes -are cumulative<a rel="footnote" href="#fn-4" name="fnd-4"><sup>4</sup></a>. - - <p>For normal builds, the Buildbot wants to get well-defined source trees -that contain specific Changes, and exclude other Changes that may have -occurred after the desired ones. We assume that the Changes arrive at -the buildbot (through one of the mechanisms described in see <a href="#Change-Sources">Change Sources</a>) in the same order in which they are committed to the -repository. The Buildbot waits for the tree to become “stable” -before initiating a build, for two reasons. The first is that -developers frequently make multiple related commits in quick -succession, even when the VC system provides ways to make atomic -transactions involving multiple files at the same time. Running a -build in the middle of these sets of changes would use an inconsistent -set of source files, and is likely to fail (and is certain to be less -useful than a build which uses the full set of changes). The -tree-stable-timer is intended to avoid these useless builds that -include some of the developer's changes but not all. The second reason -is that some VC systems (i.e. CVS) do not provide repository-wide -transaction numbers, so that timestamps are the only way to refer to -a specific repository state. These timestamps may be somewhat -ambiguous, due to processing and notification delays. By waiting until -the tree has been stable for, say, 10 minutes, we can choose a -timestamp from the middle of that period to use for our source -checkout, and then be reasonably sure that any clock-skew errors will -not cause the build to be performed on an inconsistent set of source -files. - - <p>The Schedulers always use the tree-stable-timer, with a timeout that -is configured to reflect a reasonable tradeoff between build latency -and change frequency. When the VC system provides coherent -repository-wide revision markers (such as Subversion's revision -numbers, or in fact anything other than CVS's timestamps), the -resulting Build is simply performed against a source tree defined by -that revision marker. When the VC system does not provide this, a -timestamp from the middle of the tree-stable period is used to -generate the source tree<a rel="footnote" href="#fn-5" name="fnd-5"><sup>5</sup></a>. - -<div class="node"> -<p><hr> -<a name="How-Different-VC-Systems-Specify-Sources"></a> -Next: <a rel="next" accesskey="n" href="#Attributes-of-Changes">Attributes of Changes</a>, -Previous: <a rel="previous" accesskey="p" href="#Source-Tree-Specifications">Source Tree Specifications</a>, -Up: <a rel="up" accesskey="u" href="#Version-Control-Systems">Version Control Systems</a> - -</div> - -<h4 class="subsection">3.1.3 How Different VC Systems Specify Sources</h4> - -<p>For CVS, the static specifications are <code>repository</code> and -<code>module</code>. In addition to those, each build uses a timestamp (or -omits the timestamp to mean <code>the latest</code>) and <code>branch tag</code> -(which defaults to HEAD). These parameters collectively specify a set -of sources from which a build may be performed. - - <p><a href="http://subversion.tigris.org">Subversion</a> combines the -repository, module, and branch into a single <code>Subversion URL</code> -parameter. Within that scope, source checkouts can be specified by a -numeric <code>revision number</code> (a repository-wide -monotonically-increasing marker, such that each transaction that -changes the repository is indexed by a different revision number), or -a revision timestamp. When branches are used, the repository and -module form a static <code>baseURL</code>, while each build has a -<code>revision number</code> and a <code>branch</code> (which defaults to a -statically-specified <code>defaultBranch</code>). The <code>baseURL</code> and -<code>branch</code> are simply concatenated together to derive the -<code>svnurl</code> to use for the checkout. - - <p><a href="http://www.perforce.com/">Perforce</a> is similar. The server -is specified through a <code>P4PORT</code> parameter. Module and branch -are specified in a single depot path, and revisions are -depot-wide. When branches are used, the <code>p4base</code> and -<code>defaultBranch</code> are concatenated together to produce the depot -path. - - <p><a href="http://wiki.gnuarch.org/">Arch</a> and -<a href="http://bazaar.canonical.com/">Bazaar</a> specify a repository by -URL, as well as a <code>version</code> which is kind of like a branch name. -Arch uses the word <code>archive</code> to represent the repository. Arch -lets you push changes from one archive to another, removing the strict -centralization required by CVS and SVN. It retains the distinction -between repository and working directory that most other VC systems -use. For complex multi-module directory structures, Arch has a -built-in <code>build config</code> layer with which the checkout process has -two steps. First, an initial bootstrap checkout is performed to -retrieve a set of build-config files. Second, one of these files is -used to figure out which archives/modules should be used to populate -subdirectories of the initial checkout. - - <p>Builders which use Arch and Bazaar therefore have a static archive -<code>url</code>, and a default “branch” (which is a string that specifies -a complete category–branch–version triple). Each build can have its -own branch (the category–branch–version string) to override the -default, as well as a revision number (which is turned into a -–patch-NN suffix when performing the checkout). - - <p><a href="http://bazaar-vcs.org">Bzr</a> (which is a descendant of -Arch/Bazaar, and is frequently referred to as “Bazaar”) has the same -sort of repository-vs-workspace model as Arch, but the repository data -can either be stored inside the working directory or kept elsewhere -(either on the same machine or on an entirely different machine). For -the purposes of Buildbot (which never commits changes), the repository -is specified with a URL and a revision number. - - <p>The most common way to obtain read-only access to a bzr tree is via -HTTP, simply by making the repository visible through a web server -like Apache. Bzr can also use FTP and SFTP servers, if the buildslave -process has sufficient privileges to access them. Higher performance -can be obtained by running a special Bazaar-specific server. None of -these matter to the buildbot: the repository URL just has to match the -kind of server being used. The <code>repoURL</code> argument provides the -location of the repository. - - <p>Branches are expressed as subdirectories of the main central -repository, which means that if branches are being used, the BZR step -is given a <code>baseURL</code> and <code>defaultBranch</code> instead of getting -the <code>repoURL</code> argument. - - <p><a href="http://darcs.net/">Darcs</a> doesn't really have the -notion of a single master repository. Nor does it really have -branches. In Darcs, each working directory is also a repository, and -there are operations to push and pull patches from one of these -<code>repositories</code> to another. For the Buildbot's purposes, all you -need to do is specify the URL of a repository that you want to build -from. The build slave will then pull the latest patches from that -repository and build them. Multiple branches are implemented by using -multiple repositories (possibly living on the same server). - - <p>Builders which use Darcs therefore have a static <code>repourl</code> which -specifies the location of the repository. If branches are being used, -the source Step is instead configured with a <code>baseURL</code> and a -<code>defaultBranch</code>, and the two strings are simply concatenated -together to obtain the repository's URL. Each build then has a -specific branch which replaces <code>defaultBranch</code>, or just uses the -default one. Instead of a revision number, each build can have a -“context”, which is a string that records all the patches that are -present in a given tree (this is the output of <samp><span class="command">darcs changes ---context</span></samp>, and is considerably less concise than, e.g. Subversion's -revision number, but the patch-reordering flexibility of Darcs makes -it impossible to provide a shorter useful specification). - - <p><a href="http://selenic.com/mercurial">Mercurial</a> is like Darcs, in that -each branch is stored in a separate repository. The <code>repourl</code>, -<code>baseURL</code>, and <code>defaultBranch</code> arguments are all handled the -same way as with Darcs. The “revision”, however, is the hash -identifier returned by <samp><span class="command">hg identify</span></samp>. - - <p><a href="http://git.or.cz/">Git</a> also follows a decentralized model, and -each repository can have several branches and tags. The source Step is -configured with a static <code>repourl</code> which specifies the location -of the repository. In addition, an optional <code>branch</code> parameter -can be specified to check out code from a specific branch instead of -the default “master” branch. The “revision” is specified as a SHA1 -hash as returned by e.g. <samp><span class="command">git rev-parse</span></samp>. No attempt is made -to ensure that the specified revision is actually a subset of the -specified branch. - -<div class="node"> -<p><hr> -<a name="Attributes-of-Changes"></a> -Previous: <a rel="previous" accesskey="p" href="#How-Different-VC-Systems-Specify-Sources">How Different VC Systems Specify Sources</a>, -Up: <a rel="up" accesskey="u" href="#Version-Control-Systems">Version Control Systems</a> - -</div> - -<h4 class="subsection">3.1.4 Attributes of Changes</h4> - -<h3 class="heading">Who</h3> - -<p>Each Change has a <code>who</code> attribute, which specifies which -developer is responsible for the change. This is a string which comes -from a namespace controlled by the VC repository. Frequently this -means it is a username on the host which runs the repository, but not -all VC systems require this (Arch, for example, uses a fully-qualified -<code>Arch ID</code>, which looks like an email address, as does Darcs). -Each StatusNotifier will map the <code>who</code> attribute into something -appropriate for their particular means of communication: an email -address, an IRC handle, etc. - -<h3 class="heading">Files</h3> - -<p>It also has a list of <code>files</code>, which are just the tree-relative -filenames of any files that were added, deleted, or modified for this -Change. These filenames are used by the <code>fileIsImportant</code> -function (in the Scheduler) to decide whether it is worth triggering a -new build or not, e.g. the function could use the following function -to only run a build if a C file were checked in: - -<pre class="example"> def has_C_files(change): - for name in change.files: - if name.endswith(".c"): - return True - return False -</pre> - <p>Certain BuildSteps can also use the list of changed files -to run a more targeted series of tests, e.g. the -<code>python_twisted.Trial</code> step can run just the unit tests that -provide coverage for the modified .py files instead of running the -full test suite. - -<h3 class="heading">Comments</h3> - -<p>The Change also has a <code>comments</code> attribute, which is a string -containing any checkin comments. - -<h3 class="heading">Revision</h3> - -<p>Each Change can have a <code>revision</code> attribute, which describes how -to get a tree with a specific state: a tree which includes this Change -(and all that came before it) but none that come after it. If this -information is unavailable, the <code>.revision</code> attribute will be -<code>None</code>. These revisions are provided by the ChangeSource, and -consumed by the <code>computeSourceRevision</code> method in the appropriate -<code>step.Source</code> class. - - <dl> -<dt>‘<samp><span class="samp">CVS</span></samp>’<dd><code>revision</code> is an int, seconds since the epoch -<br><dt>‘<samp><span class="samp">SVN</span></samp>’<dd><code>revision</code> is an int, the changeset number (r%d) -<br><dt>‘<samp><span class="samp">Darcs</span></samp>’<dd><code>revision</code> is a large string, the output of <code>darcs changes --context</code> -<br><dt>‘<samp><span class="samp">Mercurial</span></samp>’<dd><code>revision</code> is a short string (a hash ID), the output of <code>hg identify</code> -<br><dt>‘<samp><span class="samp">Arch/Bazaar</span></samp>’<dd><code>revision</code> is the full revision ID (ending in –patch-%d) -<br><dt>‘<samp><span class="samp">P4</span></samp>’<dd><code>revision</code> is an int, the transaction number -<br><dt>‘<samp><span class="samp">Git</span></samp>’<dd><code>revision</code> is a short string (a SHA1 hash), the output of e.g. -<code>git rev-parse</code> -</dl> - -<h3 class="heading">Branches</h3> - -<p>The Change might also have a <code>branch</code> attribute. This indicates -that all of the Change's files are in the same named branch. The -Schedulers get to decide whether the branch should be built or not. - - <p>For VC systems like CVS, Arch, Monotone, and Git, the <code>branch</code> -name is unrelated to the filename. (that is, the branch name and the -filename inhabit unrelated namespaces). For SVN, branches are -expressed as subdirectories of the repository, so the file's -“svnurl” is a combination of some base URL, the branch name, and the -filename within the branch. (In a sense, the branch name and the -filename inhabit the same namespace). Darcs branches are -subdirectories of a base URL just like SVN. Mercurial branches are the -same as Darcs. - - <dl> -<dt>‘<samp><span class="samp">CVS</span></samp>’<dd>branch='warner-newfeature', files=['src/foo.c'] -<br><dt>‘<samp><span class="samp">SVN</span></samp>’<dd>branch='branches/warner-newfeature', files=['src/foo.c'] -<br><dt>‘<samp><span class="samp">Darcs</span></samp>’<dd>branch='warner-newfeature', files=['src/foo.c'] -<br><dt>‘<samp><span class="samp">Mercurial</span></samp>’<dd>branch='warner-newfeature', files=['src/foo.c'] -<br><dt>‘<samp><span class="samp">Arch/Bazaar</span></samp>’<dd>branch='buildbot–usebranches–0', files=['buildbot/master.py'] -<br><dt>‘<samp><span class="samp">Git</span></samp>’<dd>branch='warner-newfeature', files=['src/foo.c'] -</dl> - -<h3 class="heading">Links</h3> - -<!-- TODO: who is using 'links'? how is it being used? --> -<p>Finally, the Change might have a <code>links</code> list, which is intended -to provide a list of URLs to a <em>viewcvs</em>-style web page that -provides more detail for this Change, perhaps including the full file -diffs. - -<div class="node"> -<p><hr> -<a name="Schedulers"></a> -Next: <a rel="next" accesskey="n" href="#BuildSet">BuildSet</a>, -Previous: <a rel="previous" accesskey="p" href="#Version-Control-Systems">Version Control Systems</a>, -Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a> - -</div> - -<h3 class="section">3.2 Schedulers</h3> - -<p><a name="index-Scheduler-6"></a> -Each Buildmaster has a set of <code>Scheduler</code> objects, each of which -gets a copy of every incoming Change. The Schedulers are responsible -for deciding when Builds should be run. Some Buildbot installations -might have a single Scheduler, while others may have several, each for -a different purpose. - - <p>For example, a “quick” scheduler might exist to give immediate -feedback to developers, hoping to catch obvious problems in the code -that can be detected quickly. These typically do not run the full test -suite, nor do they run on a wide variety of platforms. They also -usually do a VC update rather than performing a brand-new checkout -each time. You could have a “quick” scheduler which used a 30 second -timeout, and feeds a single “quick” Builder that uses a VC -<code>mode='update'</code> setting. - - <p>A separate “full” scheduler would run more comprehensive tests a -little while later, to catch more subtle problems. This scheduler -would have a longer tree-stable-timer, maybe 30 minutes, and would -feed multiple Builders (with a <code>mode=</code> of <code>'copy'</code>, -<code>'clobber'</code>, or <code>'export'</code>). - - <p>The <code>tree-stable-timer</code> and <code>fileIsImportant</code> decisions are -made by the Scheduler. Dependencies are also implemented here. -Periodic builds (those which are run every N seconds rather than after -new Changes arrive) are triggered by a special <code>Periodic</code> -Scheduler subclass. The default Scheduler class can also be told to -watch for specific branches, ignoring Changes on other branches. This -may be useful if you have a trunk and a few release branches which -should be tracked, but when you don't want to have the Buildbot pay -attention to several dozen private user branches. - - <p>When the setup has multiple sources of Changes the <code>category</code> -can be used for <code>Scheduler</code> objects to filter out a subset -of the Changes. Note that not all change sources can attach a category. - - <p>Some Schedulers may trigger builds for other reasons, other than -recent Changes. For example, a Scheduler subclass could connect to a -remote buildmaster and watch for builds of a library to succeed before -triggering a local build that uses that library. - - <p>Each Scheduler creates and submits <code>BuildSet</code> objects to the -<code>BuildMaster</code>, which is then responsible for making sure the -individual <code>BuildRequests</code> are delivered to the target -<code>Builders</code>. - - <p><code>Scheduler</code> instances are activated by placing them in the -<code>c['schedulers']</code> list in the buildmaster config file. Each -Scheduler has a unique name. - -<div class="node"> -<p><hr> -<a name="BuildSet"></a> -Next: <a rel="next" accesskey="n" href="#BuildRequest">BuildRequest</a>, -Previous: <a rel="previous" accesskey="p" href="#Schedulers">Schedulers</a>, -Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a> - -</div> - -<h3 class="section">3.3 BuildSet</h3> - -<p><a name="index-BuildSet-7"></a> -A <code>BuildSet</code> is the name given to a set of Builds that all -compile/test the same version of the tree on multiple Builders. In -general, all these component Builds will perform the same sequence of -Steps, using the same source code, but on different platforms or -against a different set of libraries. - - <p>The <code>BuildSet</code> is tracked as a single unit, which fails if any of -the component Builds have failed, and therefore can succeed only if -<em>all</em> of the component Builds have succeeded. There are two kinds -of status notification messages that can be emitted for a BuildSet: -the <code>firstFailure</code> type (which fires as soon as we know the -BuildSet will fail), and the <code>Finished</code> type (which fires once -the BuildSet has completely finished, regardless of whether the -overall set passed or failed). - - <p>A <code>BuildSet</code> is created with a <em>source stamp</em> tuple of -(branch, revision, changes, patch), some of which may be None, and a -list of Builders on which it is to be run. They are then given to the -BuildMaster, which is responsible for creating a separate -<code>BuildRequest</code> for each Builder. - - <p>There are a couple of different likely values for the -<code>SourceStamp</code>: - - <dl> -<dt><code>(revision=None, changes=[CHANGES], patch=None)</code><dd>This is a <code>SourceStamp</code> used when a series of Changes have -triggered a build. The VC step will attempt to check out a tree that -contains CHANGES (and any changes that occurred before CHANGES, but -not any that occurred after them). - - <br><dt><code>(revision=None, changes=None, patch=None)</code><dd>This builds the most recent code on the default branch. This is the -sort of <code>SourceStamp</code> that would be used on a Build that was -triggered by a user request, or a Periodic scheduler. It is also -possible to configure the VC Source Step to always check out the -latest sources rather than paying attention to the Changes in the -SourceStamp, which will result in same behavior as this. - - <br><dt><code>(branch=BRANCH, revision=None, changes=None, patch=None)</code><dd>This builds the most recent code on the given BRANCH. Again, this is -generally triggered by a user request or Periodic build. - - <br><dt><code>(revision=REV, changes=None, patch=(LEVEL, DIFF))</code><dd>This checks out the tree at the given revision REV, then applies a -patch (using <code>patch -pLEVEL <DIFF</code>). The <a href="#try">try</a> feature uses -this kind of <code>SourceStamp</code>. If <code>patch</code> is None, the patching -step is bypassed. - - </dl> - - <p>The buildmaster is responsible for turning the <code>BuildSet</code> into a -set of <code>BuildRequest</code> objects and queueing them on the -appropriate Builders. - -<div class="node"> -<p><hr> -<a name="BuildRequest"></a> -Next: <a rel="next" accesskey="n" href="#Builder">Builder</a>, -Previous: <a rel="previous" accesskey="p" href="#BuildSet">BuildSet</a>, -Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a> - -</div> - -<h3 class="section">3.4 BuildRequest</h3> - -<p><a name="index-BuildRequest-8"></a> -A <code>BuildRequest</code> is a request to build a specific set of sources -on a single specific <code>Builder</code>. Each <code>Builder</code> runs the -<code>BuildRequest</code> as soon as it can (i.e. when an associated -buildslave becomes free). <code>BuildRequest</code>s are prioritized from -oldest to newest, so when a buildslave becomes free, the -<code>Builder</code> with the oldest <code>BuildRequest</code> is run. - - <p>The <code>BuildRequest</code> contains the <code>SourceStamp</code> specification. -The actual process of running the build (the series of Steps that will -be executed) is implemented by the <code>Build</code> object. In this future -this might be changed, to have the <code>Build</code> define <em>what</em> -gets built, and a separate <code>BuildProcess</code> (provided by the -Builder) to define <em>how</em> it gets built. - - <p><code>BuildRequest</code> is created with optional <code>Properties</code>. One -of these, <code>owner</code>, is collected by the resultant <code>Build</code> and -added to the set of <em>interested users</em> to which status -notifications will be sent, depending on the configuration for each -status object. - - <p>The <code>BuildRequest</code> may be mergeable with other compatible -<code>BuildRequest</code>s. Builds that are triggered by incoming Changes -will generally be mergeable. Builds that are triggered by user -requests are generally not, unless they are multiple requests to build -the <em>latest sources</em> of the same branch. - -<div class="node"> -<p><hr> -<a name="Builder"></a> -Next: <a rel="next" accesskey="n" href="#Users">Users</a>, -Previous: <a rel="previous" accesskey="p" href="#BuildRequest">BuildRequest</a>, -Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a> - -</div> - -<h3 class="section">3.5 Builder</h3> - -<p><a name="index-Builder-9"></a> -The <code>Builder</code> is a long-lived object which controls all Builds of -a given type. Each one is created when the config file is first -parsed, and lives forever (or rather until it is removed from the -config file). It mediates the connections to the buildslaves that do -all the work, and is responsible for creating the <code>Build</code> objects -that decide <em>how</em> a build is performed (i.e., which steps are -executed in what order). - - <p>Each <code>Builder</code> gets a unique name, and the path name of a -directory where it gets to do all its work (there is a -buildmaster-side directory for keeping status information, as well as -a buildslave-side directory where the actual checkout/compile/test -commands are executed). It also gets a <code>BuildFactory</code>, which is -responsible for creating new <code>Build</code> instances: because the -<code>Build</code> instance is what actually performs each build, choosing -the <code>BuildFactory</code> is the way to specify what happens each time a -build is done. - - <p>Each <code>Builder</code> is associated with one of more <code>BuildSlaves</code>. -A <code>Builder</code> which is used to perform OS-X builds (as opposed to -Linux or Solaris builds) should naturally be associated with an -OS-X-based buildslave. - - <p>A <code>Builder</code> may be given a set of environment variables to be used -in its see <a href="#ShellCommand">ShellCommand</a>s. These variables will override anything in the -buildslave's environment. Variables passed directly to a ShellCommand will -override variables of the same name passed to the Builder. - - <p>For example, if you a pool of identical slaves it is often easier to manage -variables like PATH from Buildbot rather than manually editing it inside of -the slaves' environment. - -<pre class="example"> f = factory.BuildFactory - f.addStep(ShellCommand( - command=['bash', './configure'])) - f.addStep(Compile()) - - c['builders'] = [ - {'name': 'test', 'slavenames': ['slave1', 'slave2', 'slave3', 'slave4', - 'slave5', 'slave6'], - 'builddir': 'test', 'factory': f', - 'env': {'PATH': '/opt/local/bin:/opt/app/bin:/usr/local/bin:/usr/bin'}} - -</pre> - <div class="node"> -<p><hr> -<a name="Users"></a> -Next: <a rel="next" accesskey="n" href="#Build-Properties">Build Properties</a>, -Previous: <a rel="previous" accesskey="p" href="#Builder">Builder</a>, -Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a> - -</div> - -<h3 class="section">3.6 Users</h3> - -<p><a name="index-Users-10"></a> -Buildbot has a somewhat limited awareness of <em>users</em>. It assumes -the world consists of a set of developers, each of whom can be -described by a couple of simple attributes. These developers make -changes to the source code, causing builds which may succeed or fail. - - <p>Each developer is primarily known through the source control system. Each -Change object that arrives is tagged with a <code>who</code> field that -typically gives the account name (on the repository machine) of the user -responsible for that change. This string is the primary key by which the -User is known, and is displayed on the HTML status pages and in each Build's -“blamelist”. - - <p>To do more with the User than just refer to them, this username needs to -be mapped into an address of some sort. The responsibility for this mapping -is left up to the status module which needs the address. The core code knows -nothing about email addresses or IRC nicknames, just user names. - -<ul class="menu"> -<li><a accesskey="1" href="#Doing-Things-With-Users">Doing Things With Users</a> -<li><a accesskey="2" href="#Email-Addresses">Email Addresses</a> -<li><a accesskey="3" href="#IRC-Nicknames">IRC Nicknames</a> -<li><a accesskey="4" href="#Live-Status-Clients">Live Status Clients</a> -</ul> - -<div class="node"> -<p><hr> -<a name="Doing-Things-With-Users"></a> -Next: <a rel="next" accesskey="n" href="#Email-Addresses">Email Addresses</a>, -Previous: <a rel="previous" accesskey="p" href="#Users">Users</a>, -Up: <a rel="up" accesskey="u" href="#Users">Users</a> - -</div> - -<h4 class="subsection">3.6.1 Doing Things With Users</h4> - -<p>Each Change has a single User who is responsible for that Change. Most -Builds have a set of Changes: the Build represents the first time these -Changes have been built and tested by the Buildbot. The build has a -“blamelist” that consists of a simple union of the Users responsible -for all the Build's Changes. - - <p>The Build provides (through the IBuildStatus interface) a list of Users -who are “involved” in the build. For now this is equal to the -blamelist, but in the future it will be expanded to include a “build -sheriff” (a person who is “on duty” at that time and responsible for -watching over all builds that occur during their shift), as well as -per-module owners who simply want to keep watch over their domain (chosen by -subdirectory or a regexp matched against the filenames pulled out of the -Changes). The Involved Users are those who probably have an interest in the -results of any given build. - - <p>In the future, Buildbot will acquire the concept of “Problems”, -which last longer than builds and have beginnings and ends. For example, a -test case which passed in one build and then failed in the next is a -Problem. The Problem lasts until the test case starts passing again, at -which point the Problem is said to be “resolved”. - - <p>If there appears to be a code change that went into the tree at the -same time as the test started failing, that Change is marked as being -resposible for the Problem, and the user who made the change is added -to the Problem's “Guilty” list. In addition to this user, there may -be others who share responsibility for the Problem (module owners, -sponsoring developers). In addition to the Responsible Users, there -may be a set of Interested Users, who take an interest in the fate of -the Problem. - - <p>Problems therefore have sets of Users who may want to be kept aware of -the condition of the problem as it changes over time. If configured, the -Buildbot can pester everyone on the Responsible list with increasing -harshness until the problem is resolved, with the most harshness reserved -for the Guilty parties themselves. The Interested Users may merely be told -when the problem starts and stops, as they are not actually responsible for -fixing anything. - -<div class="node"> -<p><hr> -<a name="Email-Addresses"></a> -Next: <a rel="next" accesskey="n" href="#IRC-Nicknames">IRC Nicknames</a>, -Previous: <a rel="previous" accesskey="p" href="#Doing-Things-With-Users">Doing Things With Users</a>, -Up: <a rel="up" accesskey="u" href="#Users">Users</a> - -</div> - -<h4 class="subsection">3.6.2 Email Addresses</h4> - -<p>The <code>buildbot.status.mail.MailNotifier</code> class -(see <a href="#MailNotifier">MailNotifier</a>) provides a status target which can send email -about the results of each build. It accepts a static list of email -addresses to which each message should be delivered, but it can also -be configured to send mail to the Build's Interested Users. To do -this, it needs a way to convert User names into email addresses. - - <p>For many VC systems, the User Name is actually an account name on the -system which hosts the repository. As such, turning the name into an -email address is a simple matter of appending -“@repositoryhost.com”. Some projects use other kinds of mappings -(for example the preferred email address may be at “project.org” -despite the repository host being named “cvs.project.org”), and some -VC systems have full separation between the concept of a user and that -of an account on the repository host (like Perforce). Some systems -(like Arch) put a full contact email address in every change. - - <p>To convert these names to addresses, the MailNotifier uses an EmailLookup -object. This provides a .getAddress method which accepts a name and -(eventually) returns an address. The default <code>MailNotifier</code> -module provides an EmailLookup which simply appends a static string, -configurable when the notifier is created. To create more complex behaviors -(perhaps using an LDAP lookup, or using “finger” on a central host to -determine a preferred address for the developer), provide a different object -as the <code>lookup</code> argument. - - <p>In the future, when the Problem mechanism has been set up, the Buildbot -will need to send mail to arbitrary Users. It will do this by locating a -MailNotifier-like object among all the buildmaster's status targets, and -asking it to send messages to various Users. This means the User-to-address -mapping only has to be set up once, in your MailNotifier, and every email -message the buildbot emits will take advantage of it. - -<div class="node"> -<p><hr> -<a name="IRC-Nicknames"></a> -Next: <a rel="next" accesskey="n" href="#Live-Status-Clients">Live Status Clients</a>, -Previous: <a rel="previous" accesskey="p" href="#Email-Addresses">Email Addresses</a>, -Up: <a rel="up" accesskey="u" href="#Users">Users</a> - -</div> - -<h4 class="subsection">3.6.3 IRC Nicknames</h4> - -<p>Like MailNotifier, the <code>buildbot.status.words.IRC</code> class -provides a status target which can announce the results of each build. It -also provides an interactive interface by responding to online queries -posted in the channel or sent as private messages. - - <p>In the future, the buildbot can be configured map User names to IRC -nicknames, to watch for the recent presence of these nicknames, and to -deliver build status messages to the interested parties. Like -<code>MailNotifier</code> does for email addresses, the <code>IRC</code> object -will have an <code>IRCLookup</code> which is responsible for nicknames. The -mapping can be set up statically, or it can be updated by online users -themselves (by claiming a username with some kind of “buildbot: i am -user warner” commands). - - <p>Once the mapping is established, the rest of the buildbot can ask the -<code>IRC</code> object to send messages to various users. It can report on -the likelihood that the user saw the given message (based upon how long the -user has been inactive on the channel), which might prompt the Problem -Hassler logic to send them an email message instead. - -<div class="node"> -<p><hr> -<a name="Live-Status-Clients"></a> -Previous: <a rel="previous" accesskey="p" href="#IRC-Nicknames">IRC Nicknames</a>, -Up: <a rel="up" accesskey="u" href="#Users">Users</a> - -</div> - -<h4 class="subsection">3.6.4 Live Status Clients</h4> - -<p>The Buildbot also offers a PB-based status client interface which can -display real-time build status in a GUI panel on the developer's desktop. -This interface is normally anonymous, but it could be configured to let the -buildmaster know <em>which</em> developer is using the status client. The -status client could then be used as a message-delivery service, providing an -alternative way to deliver low-latency high-interruption messages to the -developer (like “hey, you broke the build”). - -<div class="node"> -<p><hr> -<a name="Build-Properties"></a> -Previous: <a rel="previous" accesskey="p" href="#Users">Users</a>, -Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a> - -</div> - -<h3 class="section">3.7 Build Properties</h3> - -<p><a name="index-Properties-11"></a> -Each build has a set of “Build Properties”, which can be used by its -BuildStep to modify their actions. These properties, in the form of -key-value pairs, provide a general framework for dynamically altering -the behavior of a build based on its circumstances. - - <p>Properties come from a number of places: - <ul> -<li>global configuration – -These properties apply to all builds. -<li>schedulers – -A scheduler can specify properties available to all the builds it -starts. -<li>buildslaves – -A buildslave can pass properties on to the builds it performs. -<li>builds – -A build automatically sets a number of properties on itself. -<li>steps – -Steps of a build can set properties that are available to subsequent -steps. In particular, source steps set a number of properties. -</ul> - - <p>Properties are very flexible, and can be used to implement all manner -of functionality. Here are some examples: - - <p>Most Source steps record the revision that they checked out in -the <code>got_revision</code> property. A later step could use this -property to specify the name of a fully-built tarball, dropped in an -easily-acessible directory for later testing. - - <p>Some projects want to perform nightly builds as well as in response -to committed changes. Such a project would run two schedulers, -both pointing to the same set of builders, but could provide an -<code>is_nightly</code> property so that steps can distinguish the nightly -builds, perhaps to run more resource-intensive tests. - - <p>Some projects have different build processes on different systems. -Rather than create a build factory for each slave, the steps can use -buildslave properties to identify the unique aspects of each slave -and adapt the build process dynamically. - -<div class="node"> -<p><hr> -<a name="Configuration"></a> -Next: <a rel="next" accesskey="n" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>, -Previous: <a rel="previous" accesskey="p" href="#Concepts">Concepts</a>, -Up: <a rel="up" accesskey="u" href="#Top">Top</a> - -</div> - -<h2 class="chapter">4 Configuration</h2> - -<p><a name="index-Configuration-12"></a> -The buildbot's behavior is defined by the “config file”, which -normally lives in the <samp><span class="file">master.cfg</span></samp> file in the buildmaster's base -directory (but this can be changed with an option to the -<code>buildbot create-master</code> command). This file completely specifies -which Builders are to be run, which slaves they should use, how -Changes should be tracked, and where the status information is to be -sent. The buildmaster's <samp><span class="file">buildbot.tac</span></samp> file names the base -directory; everything else comes from the config file. - - <p>A sample config file was installed for you when you created the -buildmaster, but you will need to edit it before your buildbot will do -anything useful. - - <p>This chapter gives an overview of the format of this file and the -various sections in it. You will need to read the later chapters to -understand how to fill in each section properly. - -<ul class="menu"> -<li><a accesskey="1" href="#Config-File-Format">Config File Format</a> -<li><a accesskey="2" href="#Loading-the-Config-File">Loading the Config File</a> -<li><a accesskey="3" href="#Testing-the-Config-File">Testing the Config File</a> -<li><a accesskey="4" href="#Defining-the-Project">Defining the Project</a> -<li><a accesskey="5" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> -<li><a accesskey="6" href="#Merging-BuildRequests">Merging BuildRequests</a> -<li><a accesskey="7" href="#Setting-the-slaveport">Setting the slaveport</a> -<li><a accesskey="8" href="#Buildslave-Specifiers">Buildslave Specifiers</a> -<li><a accesskey="9" href="#On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves">On-Demand ("Latent") Buildslaves</a> -<li><a href="#Defining-Global-Properties">Defining Global Properties</a> -<li><a href="#Defining-Builders">Defining Builders</a> -<li><a href="#Defining-Status-Targets">Defining Status Targets</a> -<li><a href="#Debug-options">Debug options</a> -</ul> - -<div class="node"> -<p><hr> -<a name="Config-File-Format"></a> -Next: <a rel="next" accesskey="n" href="#Loading-the-Config-File">Loading the Config File</a>, -Previous: <a rel="previous" accesskey="p" href="#Configuration">Configuration</a>, -Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> - -</div> - -<h3 class="section">4.1 Config File Format</h3> - -<p>The config file is, fundamentally, just a piece of Python code which -defines a dictionary named <code>BuildmasterConfig</code>, with a number of -keys that are treated specially. You don't need to know Python to do -basic configuration, though, you can just copy the syntax of the -sample file. If you <em>are</em> comfortable writing Python code, -however, you can use all the power of a full programming language to -achieve more complicated configurations. - - <p>The <code>BuildmasterConfig</code> name is the only one which matters: all -other names defined during the execution of the file are discarded. -When parsing the config file, the Buildmaster generally compares the -old configuration with the new one and performs the minimum set of -actions necessary to bring the buildbot up to date: Builders which are -not changed are left untouched, and Builders which are modified get to -keep their old event history. - - <p>Basic Python syntax: comments start with a hash character (“#”), -tuples are defined with <code>(parenthesis, pairs)</code>, arrays are -defined with <code>[square, brackets]</code>, tuples and arrays are mostly -interchangeable. Dictionaries (data structures which map “keys” to -“values”) are defined with curly braces: <code>{'key1': 'value1', -'key2': 'value2'} </code>. Function calls (and object instantiation) can use -named parameters, like <code>w = html.Waterfall(http_port=8010)</code>. - - <p>The config file starts with a series of <code>import</code> statements, -which make various kinds of Steps and Status targets available for -later use. The main <code>BuildmasterConfig</code> dictionary is created, -then it is populated with a variety of keys. These keys are broken -roughly into the following sections, each of which is documented in -the rest of this chapter: - - <ul> -<li>Project Definitions -<li>Change Sources / Schedulers -<li>Slaveport -<li>Buildslave Configuration -<li>Builders / Interlocks -<li>Status Targets -<li>Debug options -</ul> - - <p>The config file can use a few names which are placed into its namespace: - - <dl> -<dt><code>basedir</code><dd>the base directory for the buildmaster. This string has not been -expanded, so it may start with a tilde. It needs to be expanded before -use. The config file is located in -<code>os.path.expanduser(os.path.join(basedir, 'master.cfg'))</code> - - </dl> - -<div class="node"> -<p><hr> -<a name="Loading-the-Config-File"></a> -Next: <a rel="next" accesskey="n" href="#Testing-the-Config-File">Testing the Config File</a>, -Previous: <a rel="previous" accesskey="p" href="#Config-File-Format">Config File Format</a>, -Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> - -</div> - -<h3 class="section">4.2 Loading the Config File</h3> - -<p>The config file is only read at specific points in time. It is first -read when the buildmaster is launched. Once it is running, there are -various ways to ask it to reload the config file. If you are on the -system hosting the buildmaster, you can send a <code>SIGHUP</code> signal to -it: the <samp><span class="command">buildbot</span></samp> tool has a shortcut for this: - -<pre class="example"> buildbot reconfig <var>BASEDIR</var> -</pre> - <p>This command will show you all of the lines from <samp><span class="file">twistd.log</span></samp> -that relate to the reconfiguration. If there are any problems during -the config-file reload, they will be displayed in these lines. - - <p>The debug tool (<code>buildbot debugclient --master HOST:PORT</code>) has a -“Reload .cfg” button which will also trigger a reload. In the -future, there will be other ways to accomplish this step (probably a -password-protected button on the web page, as well as a privileged IRC -command). - - <p>When reloading the config file, the buildmaster will endeavor to -change as little as possible about the running system. For example, -although old status targets may be shut down and new ones started up, -any status targets that were not changed since the last time the -config file was read will be left running and untouched. Likewise any -Builders which have not been changed will be left running. If a -Builder is modified (say, the build process is changed) while a Build -is currently running, that Build will keep running with the old -process until it completes. Any previously queued Builds (or Builds -which get queued after the reconfig) will use the new process. - -<div class="node"> -<p><hr> -<a name="Testing-the-Config-File"></a> -Next: <a rel="next" accesskey="n" href="#Defining-the-Project">Defining the Project</a>, -Previous: <a rel="previous" accesskey="p" href="#Loading-the-Config-File">Loading the Config File</a>, -Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> - -</div> - -<h3 class="section">4.3 Testing the Config File</h3> - -<p>To verify that the config file is well-formed and contains no -deprecated or invalid elements, use the “checkconfig” command: - -<pre class="example"> % buildbot checkconfig master.cfg - Config file is good! -</pre> - <p>If the config file has deprecated features (perhaps because you've -upgraded the buildmaster and need to update the config file to match), -they will be announced by checkconfig. In this case, the config file -will work, but you should really remove the deprecated items and use -the recommended replacements instead: - -<pre class="example"> % buildbot checkconfig master.cfg - /usr/lib/python2.4/site-packages/buildbot/master.py:559: DeprecationWarning: c['sources'] is - deprecated as of 0.7.6 and will be removed by 0.8.0 . Please use c['change_source'] instead. - warnings.warn(m, DeprecationWarning) - Config file is good! -</pre> - <p>If the config file is simply broken, that will be caught too: - -<pre class="example"> % buildbot checkconfig master.cfg - Traceback (most recent call last): - File "/usr/lib/python2.4/site-packages/buildbot/scripts/runner.py", line 834, in doCheckConfig - ConfigLoader(configFile) - File "/usr/lib/python2.4/site-packages/buildbot/scripts/checkconfig.py", line 31, in __init__ - self.loadConfig(configFile) - File "/usr/lib/python2.4/site-packages/buildbot/master.py", line 480, in loadConfig - exec f in localDict - File "/home/warner/BuildBot/master/foolscap/master.cfg", line 90, in ? - c[bogus] = "stuff" - NameError: name 'bogus' is not defined -</pre> - <div class="node"> -<p><hr> -<a name="Defining-the-Project"></a> -Next: <a rel="next" accesskey="n" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a>, -Previous: <a rel="previous" accesskey="p" href="#Testing-the-Config-File">Testing the Config File</a>, -Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> - -</div> - -<h3 class="section">4.4 Defining the Project</h3> - -<p>There are a couple of basic settings that you use to tell the buildbot -what project it is working on. This information is used by status -reporters to let users find out more about the codebase being -exercised by this particular Buildbot installation. - -<pre class="example"> c['projectName'] = "Buildbot" - c['projectURL'] = "http://buildbot.sourceforge.net/" - c['buildbotURL'] = "http://localhost:8010/" -</pre> - <p><a name="index-c_005b_0027projectName_0027_005d-13"></a><code>projectName</code> is a short string will be used to describe the -project that this buildbot is working on. For example, it is used as -the title of the waterfall HTML page. - - <p><a name="index-c_005b_0027projectURL_0027_005d-14"></a><code>projectURL</code> is a string that gives a URL for the project as a -whole. HTML status displays will show <code>projectName</code> as a link to -<code>projectURL</code>, to provide a link from buildbot HTML pages to your -project's home page. - - <p><a name="index-c_005b_0027buildbotURL_0027_005d-15"></a>The <code>buildbotURL</code> string should point to the location where the -buildbot's internal web server (usually the <code>html.Waterfall</code> -page) is visible. This typically uses the port number set when you -create the <code>Waterfall</code> object: the buildbot needs your help to -figure out a suitable externally-visible host name. - - <p>When status notices are sent to users (either by email or over IRC), -<code>buildbotURL</code> will be used to create a URL to the specific build -or problem that they are being notified about. It will also be made -available to queriers (over IRC) who want to find out where to get -more information about this buildbot. - - <p><a name="index-c_005b_0027logCompressionLimit_0027_005d-16"></a>The <code>logCompressionLimit</code> enables bz2-compression of build logs on -disk for logs that are bigger than the given size, or disables that -completely if given <code>False</code>. The default value is 4k, which should -be a reasonable default on most file systems. This setting has no impact -on status plugins, and merely affects the required disk space on the -master for build logs. - -<div class="node"> -<p><hr> -<a name="Change-Sources-and-Schedulers"></a> -Next: <a rel="next" accesskey="n" href="#Merging-BuildRequests">Merging BuildRequests</a>, -Previous: <a rel="previous" accesskey="p" href="#Defining-the-Project">Defining the Project</a>, -Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> - -</div> - -<h3 class="section">4.5 Change Sources and Schedulers</h3> - -<p><a name="index-c_005b_0027sources_0027_005d-17"></a><a name="index-c_005b_0027change_005fsource_0027_005d-18"></a> -The <code>c['change_source']</code> key is the ChangeSource -instance<a rel="footnote" href="#fn-6" name="fnd-6"><sup>6</sup></a> that -defines how the buildmaster learns about source code changes. More -information about what goes here is available in See <a href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>. - -<pre class="example"> from buildbot.changes.pb import PBChangeSource - c['change_source'] = PBChangeSource() -</pre> - <p><a name="index-c_005b_0027schedulers_0027_005d-19"></a> -(note: in buildbot-0.7.5 and earlier, this key was named -<code>c['sources']</code>, and required a list. <code>c['sources']</code> is -deprecated as of buildbot-0.7.6 and is scheduled to be removed in a -future release). - - <p><code>c['schedulers']</code> is a list of Scheduler instances, each -of which causes builds to be started on a particular set of -Builders. The two basic Scheduler classes you are likely to start -with are <code>Scheduler</code> and <code>Periodic</code>, but you can write a -customized subclass to implement more complicated build scheduling. - - <p>Scheduler arguments -should always be specified by name (as keyword arguments), to allow -for future expansion: - -<pre class="example"> sched = Scheduler(name="quick", builderNames=['lin', 'win']) -</pre> - <p>All schedulers have several arguments in common: - - <dl> -<dt><code>name</code><dd> -Each Scheduler must have a unique name. This is used in status -displays, and is also available in the build property <code>scheduler</code>. - - <br><dt><code>builderNames</code><dd> -This is the set of builders which this scheduler should trigger, specified -as a list of names (strings). - - <br><dt><code>properties</code><dd><a name="index-Properties-20"></a> -This is a dictionary specifying properties that will be transmitted -to all builds started by this scheduler. - - </dl> - - <p>Here is a brief catalog of the available Scheduler types. All these -Schedulers are classes in <code>buildbot.scheduler</code>, and the -docstrings there are the best source of documentation on the arguments -taken by each one. - -<ul class="menu"> -<li><a accesskey="1" href="#Scheduler-Scheduler">Scheduler Scheduler</a> -<li><a accesskey="2" href="#AnyBranchScheduler">AnyBranchScheduler</a> -<li><a accesskey="3" href="#Dependent-Scheduler">Dependent Scheduler</a> -<li><a accesskey="4" href="#Periodic-Scheduler">Periodic Scheduler</a> -<li><a accesskey="5" href="#Nightly-Scheduler">Nightly Scheduler</a> -<li><a accesskey="6" href="#Try-Schedulers">Try Schedulers</a> -<li><a accesskey="7" href="#Triggerable-Scheduler">Triggerable Scheduler</a> -</ul> - -<div class="node"> -<p><hr> -<a name="Scheduler-Scheduler"></a> -Next: <a rel="next" accesskey="n" href="#AnyBranchScheduler">AnyBranchScheduler</a>, -Previous: <a rel="previous" accesskey="p" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a>, -Up: <a rel="up" accesskey="u" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> - -</div> - -<h4 class="subsection">4.5.1 Scheduler Scheduler</h4> - -<p><a name="index-buildbot_002escheduler_002eScheduler-21"></a> -This is the original and still most popular Scheduler class. It follows -exactly one branch, and starts a configurable tree-stable-timer after -each change on that branch. When the timer expires, it starts a build -on some set of Builders. The Scheduler accepts a <code>fileIsImportant</code> -function which can be used to ignore some Changes if they do not -affect any “important” files. - - <p>The arguments to this scheduler are: - - <dl> -<dt><code>name</code> -<br><dt><code>builderNames</code> -<br><dt><code>properties</code> -<br><dt><code>branch</code><dd>This Scheduler will pay attention to a single branch, ignoring Changes -that occur on other branches. Setting <code>branch</code> equal to the -special value of <code>None</code> means it should only pay attention to -the default branch. Note that <code>None</code> is a keyword, not a string, -so you want to use <code>None</code> and not <code>"None"</code>. - - <br><dt><code>treeStableTimer</code><dd>The Scheduler will wait for this many seconds before starting the -build. If new changes are made during this interval, the timer will be -restarted, so really the build will be started after a change and then -after this many seconds of inactivity. - - <br><dt><code>fileIsImportant</code><dd>A callable which takes one argument, a Change instance, and returns -<code>True</code> if the change is worth building, and <code>False</code> if -it is not. Unimportant Changes are accumulated until the build is -triggered by an important change. The default value of None means -that all Changes are important. - - <br><dt><code>categories</code><dd>A list of categories of changes that this scheduler will respond to. If this -is specified, then any non-matching changes are ignored. - - </dl> - - <p>Example: - -<pre class="example"> from buildbot import scheduler - quick = scheduler.Scheduler(name="quick", - branch=None, - treeStableTimer=60, - builderNames=["quick-linux", "quick-netbsd"]) - full = scheduler.Scheduler(name="full", - branch=None, - treeStableTimer=5*60, - builderNames=["full-linux", "full-netbsd", "full-OSX"]) - c['schedulers'] = [quick, full] -</pre> - <p>In this example, the two “quick” builders are triggered 60 seconds -after the tree has been changed. The “full” builds do not run quite -so quickly (they wait 5 minutes), so hopefully if the quick builds -fail due to a missing file or really simple typo, the developer can -discover and fix the problem before the full builds are started. Both -Schedulers only pay attention to the default branch: any changes -on other branches are ignored by these Schedulers. Each Scheduler -triggers a different set of Builders, referenced by name. - -<div class="node"> -<p><hr> -<a name="AnyBranchScheduler"></a> -Next: <a rel="next" accesskey="n" href="#Dependent-Scheduler">Dependent Scheduler</a>, -Previous: <a rel="previous" accesskey="p" href="#Scheduler-Scheduler">Scheduler Scheduler</a>, -Up: <a rel="up" accesskey="u" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> - -</div> - -<h4 class="subsection">4.5.2 AnyBranchScheduler</h4> - -<p><a name="index-buildbot_002escheduler_002eAnyBranchScheduler-22"></a> -This scheduler uses a tree-stable-timer like the default one, but -follows multiple branches at once. Each branch gets a separate timer. - - <p>The arguments to this scheduler are: - - <dl> -<dt><code>name</code> -<br><dt><code>builderNames</code> -<br><dt><code>properties</code> -<br><dt><code>branches</code><dd>This Scheduler will pay attention to any number of branches, ignoring -Changes that occur on other branches. Branches are specified just as -for the <code>Scheduler</code> class. - - <br><dt><code>treeStableTimer</code><dd>The Scheduler will wait for this many seconds before starting the -build. If new changes are made during this interval, the timer will be -restarted, so really the build will be started after a change and then -after this many seconds of inactivity. - - <br><dt><code>fileIsImportant</code><dd>A callable which takes one argument, a Change instance, and returns -<code>True</code> if the change is worth building, and <code>False</code> if -it is not. Unimportant Changes are accumulated until the build is -triggered by an important change. The default value of None means -that all Changes are important. -</dl> - -<div class="node"> -<p><hr> -<a name="Dependent-Scheduler"></a> -Next: <a rel="next" accesskey="n" href="#Periodic-Scheduler">Periodic Scheduler</a>, -Previous: <a rel="previous" accesskey="p" href="#AnyBranchScheduler">AnyBranchScheduler</a>, -Up: <a rel="up" accesskey="u" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> - -</div> - -<h4 class="subsection">4.5.3 Dependent Scheduler</h4> - -<p><a name="index-Dependent-23"></a><a name="index-Dependencies-24"></a><a name="index-buildbot_002escheduler_002eDependent-25"></a> -It is common to wind up with one kind of build which should only be -performed if the same source code was successfully handled by some -other kind of build first. An example might be a packaging step: you -might only want to produce .deb or RPM packages from a tree that was -known to compile successfully and pass all unit tests. You could put -the packaging step in the same Build as the compile and testing steps, -but there might be other reasons to not do this (in particular you -might have several Builders worth of compiles/tests, but only wish to -do the packaging once). Another example is if you want to skip the -“full” builds after a failing “quick” build of the same source -code. Or, if one Build creates a product (like a compiled library) -that is used by some other Builder, you'd want to make sure the -consuming Build is run <em>after</em> the producing one. - - <p>You can use “Dependencies” to express this relationship -to the Buildbot. There is a special kind of Scheduler named -<code>scheduler.Dependent</code> that will watch an “upstream” Scheduler -for builds to complete successfully (on all of its Builders). Each time -that happens, the same source code (i.e. the same <code>SourceStamp</code>) -will be used to start a new set of builds, on a different set of -Builders. This “downstream” scheduler doesn't pay attention to -Changes at all. It only pays attention to the upstream scheduler. - - <p>If the build fails on any of the Builders in the upstream set, -the downstream builds will not fire. Note that, for SourceStamps -generated by a ChangeSource, the <code>revision</code> is None, meaning HEAD. -If any changes are committed between the time the upstream scheduler -begins its build and the time the dependent scheduler begins its -build, then those changes will be included in the downstream build. -See the see <a href="#Triggerable-Scheduler">Triggerable Scheduler</a> for a more flexible dependency -mechanism that can avoid this problem. - - <p>The arguments to this scheduler are: - - <dl> -<dt><code>name</code> -<br><dt><code>builderNames</code> -<br><dt><code>properties</code> -<br><dt><code>upstream</code><dd>The upstream scheduler to watch. Note that this is an “instance”, -not the name of the scheduler. -</dl> - - <p>Example: - -<pre class="example"> from buildbot import scheduler - tests = scheduler.Scheduler("just-tests", None, 5*60, - ["full-linux", "full-netbsd", "full-OSX"]) - package = scheduler.Dependent("build-package", - tests, # upstream scheduler -- no quotes! - ["make-tarball", "make-deb", "make-rpm"]) - c['schedulers'] = [tests, package] -</pre> - <div class="node"> -<p><hr> -<a name="Periodic-Scheduler"></a> -Next: <a rel="next" accesskey="n" href="#Nightly-Scheduler">Nightly Scheduler</a>, -Previous: <a rel="previous" accesskey="p" href="#Dependent-Scheduler">Dependent Scheduler</a>, -Up: <a rel="up" accesskey="u" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> - -</div> - -<h4 class="subsection">4.5.4 Periodic Scheduler</h4> - -<p><a name="index-buildbot_002escheduler_002ePeriodic-26"></a> -This simple scheduler just triggers a build every N seconds. - - <p>The arguments to this scheduler are: - - <dl> -<dt><code>name</code> -<br><dt><code>builderNames</code> -<br><dt><code>properties</code> -<br><dt><code>periodicBuildTimer</code><dd>The time, in seconds, after which to start a build. -</dl> - - <p>Example: - -<pre class="example"> from buildbot import scheduler - nightly = scheduler.Periodic(name="nightly", - builderNames=["full-solaris"], - periodicBuildTimer=24*60*60) - c['schedulers'] = [nightly] -</pre> - <p>The Scheduler in this example just runs the full solaris build once -per day. Note that this Scheduler only lets you control the time -between builds, not the absolute time-of-day of each Build, so this -could easily wind up a “daily” or “every afternoon” scheduler -depending upon when it was first activated. - -<div class="node"> -<p><hr> -<a name="Nightly-Scheduler"></a> -Next: <a rel="next" accesskey="n" href="#Try-Schedulers">Try Schedulers</a>, -Previous: <a rel="previous" accesskey="p" href="#Periodic-Scheduler">Periodic Scheduler</a>, -Up: <a rel="up" accesskey="u" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> - -</div> - -<h4 class="subsection">4.5.5 Nightly Scheduler</h4> - -<p><a name="index-buildbot_002escheduler_002eNightly-27"></a> -This is highly configurable periodic build scheduler, which triggers -a build at particular times of day, week, month, or year. The -configuration syntax is very similar to the well-known <code>crontab</code> -format, in which you provide values for minute, hour, day, and month -(some of which can be wildcards), and a build is triggered whenever -the current time matches the given constraints. This can run a build -every night, every morning, every weekend, alternate Thursdays, -on your boss's birthday, etc. - - <p>Pass some subset of <code>minute</code>, <code>hour</code>, <code>dayOfMonth</code>, -<code>month</code>, and <code>dayOfWeek</code>; each may be a single number or -a list of valid values. The builds will be triggered whenever the -current time matches these values. Wildcards are represented by a -'*' string. All fields default to a wildcard except 'minute', so -with no fields this defaults to a build every hour, on the hour. -The full list of parameters is: - - <dl> -<dt><code>name</code> -<br><dt><code>builderNames</code> -<br><dt><code>properties</code> -<br><dt><code>branch</code><dd>The branch to build, just as for <code>Scheduler</code>. - - <br><dt><code>minute</code><dd>The minute of the hour on which to start the build. This defaults -to 0, meaning an hourly build. - - <br><dt><code>hour</code><dd>The hour of the day on which to start the build, in 24-hour notation. -This defaults to *, meaning every hour. - - <br><dt><code>month</code><dd>The month in which to start the build, with January = 1. This defaults -to *, meaning every month. - - <br><dt><code>dayOfWeek</code><dd>The day of the week to start a build, with Monday = 0. This defauls -to *, meaning every day of the week. - - <br><dt><code>onlyIfChanged</code><dd>If this is true, then builds will not be scheduled at the designated time -unless the source has changed since the previous build. -</dl> - - <p>For example, the following master.cfg clause will cause a build to be -started every night at 3:00am: - -<pre class="example"> s = scheduler.Nightly(name='nightly', - builderNames=['builder1', 'builder2'], - hour=3, - minute=0) -</pre> - <p>This scheduler will perform a build each monday morning at 6:23am and -again at 8:23am, but only if someone has committed code in the interim: - -<pre class="example"> s = scheduler.Nightly(name='BeforeWork', - builderNames=['builder1'], - dayOfWeek=0, - hour=[6,8], - minute=23, - onlyIfChanged=True) -</pre> - <p>The following runs a build every two hours, using Python's <code>range</code> -function: - -<pre class="example"> s = Nightly(name='every2hours', - builderNames=['builder1'], - hour=range(0, 24, 2)) -</pre> - <p>Finally, this example will run only on December 24th: - -<pre class="example"> s = Nightly(name='SleighPreflightCheck', - builderNames=['flying_circuits', 'radar'], - month=12, - dayOfMonth=24, - hour=12, - minute=0) -</pre> - <div class="node"> -<p><hr> -<a name="Try-Schedulers"></a> -Next: <a rel="next" accesskey="n" href="#Triggerable-Scheduler">Triggerable Scheduler</a>, -Previous: <a rel="previous" accesskey="p" href="#Nightly-Scheduler">Nightly Scheduler</a>, -Up: <a rel="up" accesskey="u" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> - -</div> - -<h4 class="subsection">4.5.6 Try Schedulers</h4> - -<p><a name="index-buildbot_002escheduler_002eTry_005fJobdir-28"></a><a name="index-buildbot_002escheduler_002eTry_005fUserpass-29"></a> -This scheduler allows developers to use the <code>buildbot try</code> -command to trigger builds of code they have not yet committed. See -<a href="#try">try</a> for complete details. - - <p>Two implementations are available: <code>Try_Jobdir</code> and -<code>Try_Userpass</code>. The former monitors a job directory, specified -by the <code>jobdir</code> parameter, while the latter listens for PB -connections on a specific <code>port</code>, and authenticates against -<code>userport</code>. - -<div class="node"> -<p><hr> -<a name="Triggerable-Scheduler"></a> -Previous: <a rel="previous" accesskey="p" href="#Try-Schedulers">Try Schedulers</a>, -Up: <a rel="up" accesskey="u" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> - -</div> - -<h4 class="subsection">4.5.7 Triggerable Scheduler</h4> - -<p><a name="index-Triggers-30"></a><a name="index-buildbot_002escheduler_002eTriggerable-31"></a> -The <code>Triggerable</code> scheduler waits to be triggered by a Trigger -step (see <a href="#Triggering-Schedulers">Triggering Schedulers</a>) in another build. That step -can optionally wait for the scheduler's builds to complete. This -provides two advantages over Dependent schedulers. First, the same -scheduler can be triggered from multiple builds. Second, the ability -to wait for a Triggerable's builds to complete provides a form of -"subroutine call", where one or more builds can "call" a scheduler -to perform some work for them, perhaps on other buildslaves. - - <p>The parameters are just the basics: - - <dl> -<dt><code>name</code><br><dt><code>builderNames</code><br><dt><code>properties</code><dd></dl> - - <p>This class is only useful in conjunction with the <code>Trigger</code> step. -Here is a fully-worked example: - -<pre class="example"> from buildbot import scheduler - from buildbot.process import factory - from buildbot.steps import trigger - - checkin = scheduler.Scheduler(name="checkin", - branch=None, - treeStableTimer=5*60, - builderNames=["checkin"]) - nightly = scheduler.Nightly(name='nightly', - builderNames=['nightly'], - hour=3, - minute=0) - - mktarball = scheduler.Triggerable(name="mktarball", - builderNames=["mktarball"]) - build = scheduler.Triggerable(name="build-all-platforms", - builderNames=["build-all-platforms"]) - test = scheduler.Triggerable(name="distributed-test", - builderNames=["distributed-test"]) - package = scheduler.Triggerable(name="package-all-platforms", - builderNames=["package-all-platforms"]) - - c['schedulers'] = [checkin, nightly, build, test, package] - - # on checkin, make a tarball, build it, and test it - checkin_factory = factory.BuildFactory() - checkin_factory.addStep(trigger.Trigger(schedulerNames=['mktarball'], - waitForFinish=True)) - checkin_factory.addStep(trigger.Trigger(schedulerNames=['build-all-platforms'], - waitForFinish=True)) - checkin_factory.addStep(trigger.Trigger(schedulerNames=['distributed-test'], - waitForFinish=True)) - - # and every night, make a tarball, build it, and package it - nightly_factory = factory.BuildFactory() - nightly_factory.addStep(trigger.Trigger(schedulerNames=['mktarball'], - waitForFinish=True)) - nightly_factory.addStep(trigger.Trigger(schedulerNames=['build-all-platforms'], - waitForFinish=True)) - nightly_factory.addStep(trigger.Trigger(schedulerNames=['package-all-platforms'], - waitForFinish=True)) -</pre> - <div class="node"> -<p><hr> -<a name="Merging-BuildRequests"></a> -Next: <a rel="next" accesskey="n" href="#Setting-the-slaveport">Setting the slaveport</a>, -Previous: <a rel="previous" accesskey="p" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a>, -Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> - -</div> - -<h3 class="section">4.6 Merging BuildRequests</h3> - -<p><a name="index-c_005b_0027mergeRequests_0027_005d-32"></a> -By default, buildbot merges BuildRequests that have the compatible -SourceStamps. This behaviour can be customized with the -<code>c['mergeRequests']</code> configuration key. This key specifies a function -which is caleld with three arguments: a <code>Builder</code> and two -<code>BuildRequest</code> objects. It should return true if the requests can be -merged. For example: - -<pre class="example"> def mergeRequests(builder, req1, req2): - """Don't merge buildrequest at all""" - return False - c['mergeRequests'] = mergeRequests -</pre> - <p>In many cases, the details of the SourceStamps and BuildRequests are important. -In this example, only BuildRequests with the same "reason" are merged; thus -developers forcing builds for different reasons will see distinct builds. - -<pre class="example"> def mergeRequests(builder, req1, req2): - if req1.source.canBeMergedWith(req2.source) and req1.reason == req2.reason: - return True - return False - c['mergeRequests'] = mergeRequests -</pre> - <div class="node"> -<p><hr> -<a name="Setting-the-slaveport"></a> -Next: <a rel="next" accesskey="n" href="#Buildslave-Specifiers">Buildslave Specifiers</a>, -Previous: <a rel="previous" accesskey="p" href="#Merging-BuildRequests">Merging BuildRequests</a>, -Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> - -</div> - -<h3 class="section">4.7 Setting the slaveport</h3> - -<p><a name="index-c_005b_0027slavePortnum_0027_005d-33"></a> -The buildmaster will listen on a TCP port of your choosing for -connections from buildslaves. It can also use this port for -connections from remote Change Sources, status clients, and debug -tools. This port should be visible to the outside world, and you'll -need to tell your buildslave admins about your choice. - - <p>It does not matter which port you pick, as long it is externally -visible, however you should probably use something larger than 1024, -since most operating systems don't allow non-root processes to bind to -low-numbered ports. If your buildmaster is behind a firewall or a NAT -box of some sort, you may have to configure your firewall to permit -inbound connections to this port. - -<pre class="example"> c['slavePortnum'] = 10000 -</pre> - <p><code>c['slavePortnum']</code> is a <em>strports</em> specification string, -defined in the <code>twisted.application.strports</code> module (try -<samp><span class="command">pydoc twisted.application.strports</span></samp> to get documentation on -the format). This means that you can have the buildmaster listen on a -localhost-only port by doing: - -<pre class="example"> c['slavePortnum'] = "tcp:10000:interface=127.0.0.1" -</pre> - <p>This might be useful if you only run buildslaves on the same machine, -and they are all configured to contact the buildmaster at -<code>localhost:10000</code>. - -<div class="node"> -<p><hr> -<a name="Buildslave-Specifiers"></a> -Next: <a rel="next" accesskey="n" href="#On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves">On-Demand ("Latent") Buildslaves</a>, -Previous: <a rel="previous" accesskey="p" href="#Setting-the-slaveport">Setting the slaveport</a>, -Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> - -</div> - -<h3 class="section">4.8 Buildslave Specifiers</h3> - -<p><a name="index-c_005b_0027slaves_0027_005d-34"></a> -The <code>c['slaves']</code> key is a list of known buildslaves. In the common case, -each buildslave is defined by an instance of the BuildSlave class. It -represents a standard, manually started machine that will try to connect to -the buildbot master as a slave. Contrast these with the "on-demand" latent -buildslaves, such as the Amazon Web Service Elastic Compute Cloud latent -buildslave discussed below. - - <p>The BuildSlave class is instantiated with two values: (slavename, -slavepassword). These are the same two values that need to be provided to the -buildslave administrator when they create the buildslave. - - <p>The slavenames must be unique, of course. The password exists to -prevent evildoers from interfering with the buildbot by inserting -their own (broken) buildslaves into the system and thus displacing the -real ones. - - <p>Buildslaves with an unrecognized slavename or a non-matching password -will be rejected when they attempt to connect, and a message -describing the problem will be put in the log file (see <a href="#Logfiles">Logfiles</a>). - -<pre class="example"> from buildbot.buildslave import BuildSlave - c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd') - BuildSlave('bot-bsd', 'bsdpasswd') - ] -</pre> - <p><a name="index-Properties-35"></a><code>BuildSlave</code> objects can also be created with an optional -<code>properties</code> argument, a dictionary specifying properties that -will be available to any builds performed on this slave. For example: - -<pre class="example"> from buildbot.buildslave import BuildSlave - c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd', - properties={'os':'solaris'}), - ] -</pre> - <p>The <code>BuildSlave</code> constructor can also take an optional -<code>max_builds</code> parameter to limit the number of builds that it -will execute simultaneously: - -<pre class="example"> from buildbot.buildslave import BuildSlave - c['slaves'] = [BuildSlave("bot-linux", "linuxpassword", max_builds=2)] -</pre> - <p>Historical note: in buildbot-0.7.5 and earlier, the <code>c['bots']</code> -key was used instead, and it took a list of (name, password) tuples. -This key is accepted for backwards compatibility, but is deprecated as -of 0.7.6 and will go away in some future release. - -<ul class="menu"> -<li><a accesskey="1" href="#When-Buildslaves-Go-Missing">When Buildslaves Go Missing</a> -</ul> - -<div class="node"> -<p><hr> -<a name="When-Buildslaves-Go-Missing"></a> -Up: <a rel="up" accesskey="u" href="#Buildslave-Specifiers">Buildslave Specifiers</a> - -</div> - -<h4 class="subsection">4.8.1 When Buildslaves Go Missing</h4> - -<p>Sometimes, the buildslaves go away. One very common reason for this is -when the buildslave process is started once (manually) and left -running, but then later the machine reboots and the process is not -automatically restarted. - - <p>If you'd like to have the administrator of the buildslave (or other -people) be notified by email when the buildslave has been missing for -too long, just add the <code>notify_on_missing=</code> argument to the -<code>BuildSlave</code> definition: - -<pre class="example"> c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd', - notify_on_missing="bob@example.com"), - ] -</pre> - <p>By default, this will send email when the buildslave has been -disconnected for more than one hour. Only one email per -connection-loss event will be sent. To change the timeout, use -<code>missing_timeout=</code> and give it a number of seconds (the default -is 3600). - - <p>You can have the buildmaster send email to multiple recipients: just -provide a list of addresses instead of a single one: - -<pre class="example"> c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd', - notify_on_missing=["bob@example.com", - "alice@example.org"], - missing_timeout=300, # notify after 5 minutes - ), - ] -</pre> - <p>The email sent this way will use a MailNotifier (see <a href="#MailNotifier">MailNotifier</a>) -status target, if one is configured. This provides a way for you to -control the “from” address of the email, as well as the relayhost -(aka “smarthost”) to use as an SMTP server. If no MailNotifier is -configured on this buildmaster, the buildslave-missing emails will be -sent using a default configuration. - - <p>Note that if you want to have a MailNotifier for buildslave-missing -emails but not for regular build emails, just create one with -builders=[], as follows: - -<pre class="example"> from buildbot.status import mail - m = mail.MailNotifier(fromaddr="buildbot@localhost", builders=[], - relayhost="smtp.example.org") - c['status'].append(m) - c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd', - notify_on_missing="bob@example.com"), - ] -</pre> - <div class="node"> -<p><hr> -<a name="On-Demand-(%22Latent%22)-Buildslaves"></a> -<a name="On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves"></a> -Next: <a rel="next" accesskey="n" href="#Defining-Global-Properties">Defining Global Properties</a>, -Previous: <a rel="previous" accesskey="p" href="#Buildslave-Specifiers">Buildslave Specifiers</a>, -Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> - -</div> - -<h3 class="section">4.9 On-Demand ("Latent") Buildslaves</h3> - -<p>The standard buildbot model has slaves started manually. The previous section -described how to configure the master for this approach. - - <p>Another approach is to let the buildbot master start slaves when builds are -ready, on-demand. Thanks to services such as Amazon Web Services' Elastic -Compute Cloud ("AWS EC2"), this is relatively easy to set up, and can be -very useful for some situations. - - <p>The buildslaves that are started on-demand are called "latent" buildslaves. -As of this writing, buildbot ships with an abstract base class for building -latent buildslaves, and a concrete implementation for AWS EC2. - -<ul class="menu"> -<li><a accesskey="1" href="#Amazon-Web-Services-Elastic-Compute-Cloud-_0028_0022AWS-EC2_0022_0029">Amazon Web Services Elastic Compute Cloud ("AWS EC2")</a> -<li><a accesskey="2" href="#Dangers-with-Latent-Buildslaves">Dangers with Latent Buildslaves</a> -<li><a accesskey="3" href="#Writing-New-Latent-Buildslaves">Writing New Latent Buildslaves</a> -</ul> - -<div class="node"> -<p><hr> -<a name="Amazon-Web-Services-Elastic-Compute-Cloud-(%22AWS-EC2%22)"></a> -<a name="Amazon-Web-Services-Elastic-Compute-Cloud-_0028_0022AWS-EC2_0022_0029"></a> -Next: <a rel="next" accesskey="n" href="#Dangers-with-Latent-Buildslaves">Dangers with Latent Buildslaves</a>, -Up: <a rel="up" accesskey="u" href="#On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves">On-Demand ("Latent") Buildslaves</a> - -</div> - -<h4 class="subsection">4.9.1 Amazon Web Services Elastic Compute Cloud ("AWS EC2")</h4> - -<p><a href="http://aws.amazon.com/ec2/">AWS EC2</a> is a web service that allows you to -start virtual machines in an Amazon data center. Please see their website for -details, incuding costs. Using the AWS EC2 latent buildslaves involves getting -an EC2 account with AWS and setting up payment; customizing one or more EC2 -machine images ("AMIs") on your desired operating system(s) and publishing -them (privately if needed); and configuring the buildbot master to know how to -start your customized images for "substantiating" your latent slaves. - -<ul class="menu"> -<li><a accesskey="1" href="#Get-an-AWS-EC2-Account">Get an AWS EC2 Account</a> -<li><a accesskey="2" href="#Create-an-AMI">Create an AMI</a> -<li><a accesskey="3" href="#Configure-the-Master-with-an-EC2LatentBuildSlave">Configure the Master with an EC2LatentBuildSlave</a> -</ul> - -<div class="node"> -<p><hr> -<a name="Get-an-AWS-EC2-Account"></a> -Next: <a rel="next" accesskey="n" href="#Create-an-AMI">Create an AMI</a>, -Up: <a rel="up" accesskey="u" href="#Amazon-Web-Services-Elastic-Compute-Cloud-_0028_0022AWS-EC2_0022_0029">Amazon Web Services Elastic Compute Cloud ("AWS EC2")</a> - -</div> - -<h5 class="subsubsection">4.9.1.1 Get an AWS EC2 Account</h5> - -<p>To start off, to use the AWS EC2 latent buildslave, you need to get an AWS -developer account and sign up for EC2. These instructions may help you get -started: - - <ul> -<li>Go to http://aws.amazon.com/ and click to "Sign Up Now" for an AWS account. - - <li>Once you are logged into your account, you need to sign up for EC2. -Instructions for how to do this have changed over time because Amazon changes -their website, so the best advice is to hunt for it. After signing up for EC2, -it may say it wants you to upload an x.509 cert. You will need this to create -images (see below) but it is not technically necessary for the buildbot master -configuration. - - <li>You must enter a valid credit card before you will be able to use EC2. Do that -under 'Payment Method'. - - <li>Make sure you're signed up for EC2 by going to 'Your Account'->'Account -Activity' and verifying EC2 is listed. -</ul> - -<div class="node"> -<p><hr> -<a name="Create-an-AMI"></a> -Next: <a rel="next" accesskey="n" href="#Configure-the-Master-with-an-EC2LatentBuildSlave">Configure the Master with an EC2LatentBuildSlave</a>, -Previous: <a rel="previous" accesskey="p" href="#Get-an-AWS-EC2-Account">Get an AWS EC2 Account</a>, -Up: <a rel="up" accesskey="u" href="#Amazon-Web-Services-Elastic-Compute-Cloud-_0028_0022AWS-EC2_0022_0029">Amazon Web Services Elastic Compute Cloud ("AWS EC2")</a> - -</div> - -<h5 class="subsubsection">4.9.1.2 Create an AMI</h5> - -<p>Now you need to create an AMI and configure the master. You may need to -run through this cycle a few times to get it working, but these instructions -should get you started. - - <p>Creating an AMI is out of the scope of this document. The -<a href="http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/">EC2 Getting Started Guide</a> -is a good resource for this task. Here are a few additional hints. - - <ul> -<li>When an instance of the image starts, it needs to automatically start a -buildbot slave that connects to your master (to create a buildbot slave, -see <a href="#Creating-a-buildslave">Creating a buildslave</a>; to make a daemon, -see <a href="#Launching-the-daemons">Launching the daemons</a>). - - <li>You may want to make an instance of the buildbot slave, configure it as a -standard buildslave in the master (i.e., not as a latent slave), and test and -debug it that way before you turn it into an AMI and convert to a latent -slave in the master. -</ul> - -<div class="node"> -<p><hr> -<a name="Configure-the-Master-with-an-EC2LatentBuildSlave"></a> -Previous: <a rel="previous" accesskey="p" href="#Create-an-AMI">Create an AMI</a>, -Up: <a rel="up" accesskey="u" href="#Amazon-Web-Services-Elastic-Compute-Cloud-_0028_0022AWS-EC2_0022_0029">Amazon Web Services Elastic Compute Cloud ("AWS EC2")</a> - -</div> - -<h5 class="subsubsection">4.9.1.3 Configure the Master with an EC2LatentBuildSlave</h5> - -<p>Now let's assume you have an AMI that should work with the -EC2LatentBuildSlave. It's now time to set up your buildbot master -configuration. - - <p>You will need some information from your AWS account: the "Access Key Id" and -the "Secret Access Key". If you've built the AMI yourself, you probably -already are familiar with these values. If you have not, and someone has -given you access to an AMI, these hints may help you find the necessary -values: - - <ul> -<li>While logged into your AWS account, find the "Access Identifiers" link (either -on the left, or via "Your Account" -> "Access Identifiers". - - <li>On the page, you'll see alphanumeric values for "Your Access Key Id:" and -"Your Secret Access Key:". Make a note of these. Later on, we'll call the -first one your "identifier" and the second one your "secret_identifier." -</ul> - - <p>When creating an EC2LatentBuildSlave in the buildbot master configuration, -the first three arguments are required. The name and password are the first -two arguments, and work the same as with normal buildslaves. The next -argument specifies the type of the EC2 virtual machine (available options as -of this writing include "m1.small", "m1.large", 'm1.xlarge", "c1.medium", -and "c1.xlarge"; see the EC2 documentation for descriptions of these -machines). - - <p>Here is the simplest example of configuring an EC2 latent buildslave. It -specifies all necessary remaining values explicitly in the instantiation. - -<pre class="example"> from buildbot.ec2buildslave import EC2LatentBuildSlave - c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', - ami='ami-12345', - identifier='publickey', - secret_identifier='privatekey' - )] -</pre> - <p>The "ami" argument specifies the AMI that the master should start. The -"identifier" argument specifies the AWS "Access Key Id," and the -"secret_identifier" specifies the AWS "Secret Access Key." Both the AMI and -the account information can be specified in alternate ways. - - <p>Note that whoever has your identifier and secret_identifier values can request -AWS work charged to your account, so these values need to be carefully -protected. Another way to specify these access keys is to put them in a -separate file. You can then make the access privileges stricter for this -separate file, and potentially let more people read your main configuration -file. - - <p>By default, you can make an .ec2 directory in the home folder of the user -running the buildbot master. In that directory, create a file called aws_id. -The first line of that file should be your access key id; the second line -should be your secret access key id. Then you can instantiate the build slave -as follows. - -<pre class="example"> from buildbot.ec2buildslave import EC2LatentBuildSlave - c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', - ami='ami-12345')] -</pre> - <p>If you want to put the key information in another file, use the -"aws_id_file_path" initialization argument. - - <p>Previous examples used a particular AMI. If the Buildbot master will be -deployed in a process-controlled environment, it may be convenient to -specify the AMI more flexibly. Rather than specifying an individual AMI, -specify one or two AMI filters. - - <p>In all cases, the AMI that sorts last by its location (the S3 bucket and -manifest name) will be preferred. - - <p>One available filter is to specify the acceptable AMI owners, by AWS account -number (the 12 digit number, usually rendered in AWS with hyphens like -"1234-5678-9012", should be entered as in integer). - -<pre class="example"> from buildbot.ec2buildslave import EC2LatentBuildSlave - bot1 = EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', - valid_ami_owners=[11111111111, - 22222222222], - identifier='publickey', - secret_identifier='privatekey' - ) -</pre> - <p>The other available filter is to provide a regular expression string that -will be matched against each AMI's location (the S3 bucket and manifest name). - -<pre class="example"> from buildbot.ec2buildslave import EC2LatentBuildSlave - bot1 = EC2LatentBuildSlave( - 'bot1', 'sekrit', 'm1.large', - valid_ami_location_regex=r'buildbot\-.*/image.manifest.xml', - identifier='publickey', secret_identifier='privatekey') -</pre> - <p>The regular expression can specify a group, which will be preferred for the -sorting. Only the first group is used; subsequent groups are ignored. - -<pre class="example"> from buildbot.ec2buildslave import EC2LatentBuildSlave - bot1 = EC2LatentBuildSlave( - 'bot1', 'sekrit', 'm1.large', - valid_ami_location_regex=r'buildbot\-.*\-(.*)/image.manifest.xml', - identifier='publickey', secret_identifier='privatekey') -</pre> - <p>If the group can be cast to an integer, it will be. This allows 10 to sort -after 1, for instance. - -<pre class="example"> from buildbot.ec2buildslave import EC2LatentBuildSlave - bot1 = EC2LatentBuildSlave( - 'bot1', 'sekrit', 'm1.large', - valid_ami_location_regex=r'buildbot\-.*\-(\d+)/image.manifest.xml', - identifier='publickey', secret_identifier='privatekey') -</pre> - <p>In addition to using the password as a handshake between the master and the -slave, you may want to use a firewall to assert that only machines from a -specific IP can connect as slaves. This is possible with AWS EC2 by using -the Elastic IP feature. To configure, generate a Elastic IP in AWS, and then -specify it in your configuration using the "elastic_ip" argument. - -<pre class="example"> from buildbot.ec2buildslave import EC2LatentBuildSlave - c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', - 'ami-12345', - identifier='publickey', - secret_identifier='privatekey', - elastic_ip='208.77.188.166' - )] -</pre> - <p>The EC2LatentBuildSlave supports all other configuration from the standard -BuildSlave. The "missing_timeout" and "notify_on_missing" specify how long -to wait for an EC2 instance to attach before considering the attempt to have -failed, and email addresses to alert, respectively. "missing_timeout" -defaults to 20 minutes. - - <p>The "build_wait_timeout" allows you to specify how long an EC2LatentBuildSlave -should wait after a build for another build before it shuts down the EC2 -instance. It defaults to 10 minutes. - - <p>"keypair_name" and "security_name" allow you to specify different names for -these AWS EC2 values. They both default to "latent_buildbot_slave". - -<div class="node"> -<p><hr> -<a name="Dangers-with-Latent-Buildslaves"></a> -Next: <a rel="next" accesskey="n" href="#Writing-New-Latent-Buildslaves">Writing New Latent Buildslaves</a>, -Previous: <a rel="previous" accesskey="p" href="#Amazon-Web-Services-Elastic-Compute-Cloud-_0028_0022AWS-EC2_0022_0029">Amazon Web Services Elastic Compute Cloud ("AWS EC2")</a>, -Up: <a rel="up" accesskey="u" href="#On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves">On-Demand ("Latent") Buildslaves</a> - -</div> - -<h4 class="subsection">4.9.2 Dangers with Latent Buildslaves</h4> - -<p>Any latent build slave that interacts with a for-fee service, such as the -EC2LatentBuildSlave, brings significant risks. As already identified, the -configuraton will need access to account information that, if obtained by a -criminal, can be used to charge services to your account. Also, bugs in the -buildbot software may lead to unnecessary charges. In particular, if the -master neglects to shut down an instance for some reason, a virtual machine -may be running unnecessarily, charging against your account. Manual and/or -automatic (e.g. nagios with a plugin using a library like boto) -double-checking may be appropriate. - - <p>A comparitively trivial note is that currently if two instances try to attach -to the same latent buildslave, it is likely that the system will become -confused. This should not occur, unless, for instance, you configure a normal -build slave to connect with the authentication of a latent buildbot. If the -situation occurs, stop all attached instances and restart the master. - -<div class="node"> -<p><hr> -<a name="Writing-New-Latent-Buildslaves"></a> -Previous: <a rel="previous" accesskey="p" href="#Dangers-with-Latent-Buildslaves">Dangers with Latent Buildslaves</a>, -Up: <a rel="up" accesskey="u" href="#On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves">On-Demand ("Latent") Buildslaves</a> - -</div> - -<h4 class="subsection">4.9.3 Writing New Latent Buildslaves</h4> - -<p>Writing a new latent buildslave should only require subclassing -<code>buildbot.buildslave.AbstractLatentBuildSlave</code> and implementing -start_instance and stop_instance. - -<pre class="example"> def start_instance(self): - # responsible for starting instance that will try to connect with this - # master. Should return deferred. Problems should use an errback. The - # callback value can be None, or can be an iterable of short strings to - # include in the "substantiate success" status message, such as - # identifying the instance that started. - raise NotImplementedError - - def stop_instance(self, fast=False): - # responsible for shutting down instance. Return a deferred. If `fast`, - # we're trying to shut the master down, so callback as soon as is safe. - # Callback value is ignored. - raise NotImplementedError -</pre> - <p>See <code>buildbot.ec2buildslave.EC2LatentBuildSlave</code> for an example, or see the -test example <code>buildbot.test_slaves.FakeLatentBuildSlave</code>. - -<div class="node"> -<p><hr> -<a name="Defining-Global-Properties"></a> -Next: <a rel="next" accesskey="n" href="#Defining-Builders">Defining Builders</a>, -Previous: <a rel="previous" accesskey="p" href="#On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves">On-Demand ("Latent") Buildslaves</a>, -Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> - -</div> - -<h3 class="section">4.10 Defining Global Properties</h3> - -<p><a name="index-c_005b_0027properties_0027_005d-36"></a><a name="index-Properties-37"></a> -The <code>'properties'</code> configuration key defines a dictionary -of properties that will be available to all builds started by the -buildmaster: - -<pre class="example"> c['properties'] = { - 'Widget-version' : '1.2', - 'release-stage' : 'alpha' - } -</pre> - <div class="node"> -<p><hr> -<a name="Defining-Builders"></a> -Next: <a rel="next" accesskey="n" href="#Defining-Status-Targets">Defining Status Targets</a>, -Previous: <a rel="previous" accesskey="p" href="#Defining-Global-Properties">Defining Global Properties</a>, -Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> - -</div> - -<h3 class="section">4.11 Defining Builders</h3> - -<p><a name="index-c_005b_0027builders_0027_005d-38"></a> -The <code>c['builders']</code> key is a list of dictionaries which specify -the Builders. The Buildmaster runs a collection of Builders, each of -which handles a single type of build (e.g. full versus quick), on a -single build slave. A Buildbot which makes sure that the latest code -(“HEAD”) compiles correctly across four separate architecture will -have four Builders, each performing the same build but on different -slaves (one per platform). - - <p>Each Builder gets a separate column in the waterfall display. In -general, each Builder runs independently (although various kinds of -interlocks can cause one Builder to have an effect on another). - - <p>Each Builder specification dictionary has several required keys: - - <dl> -<dt><code>name</code><dd>This specifies the Builder's name, which is used in status -reports. - - <br><dt><code>slavename</code><dd>This specifies which buildslave will be used by this Builder. -<code>slavename</code> must appear in the <code>c['slaves']</code> list. Each -buildslave can accomodate multiple Builders. - - <br><dt><code>slavenames</code><dd>If you provide <code>slavenames</code> instead of <code>slavename</code>, you can -give a list of buildslaves which are capable of running this Builder. -If multiple buildslaves are available for any given Builder, you will -have some measure of redundancy: in case one slave goes offline, the -others can still keep the Builder working. In addition, multiple -buildslaves will allow multiple simultaneous builds for the same -Builder, which might be useful if you have a lot of forced or “try” -builds taking place. - - <p>If you use this feature, it is important to make sure that the -buildslaves are all, in fact, capable of running the given build. The -slave hosts should be configured similarly, otherwise you will spend a -lot of time trying (unsuccessfully) to reproduce a failure that only -occurs on some of the buildslaves and not the others. Different -platforms, operating systems, versions of major programs or libraries, -all these things mean you should use separate Builders. - - <br><dt><code>builddir</code><dd>This specifies the name of a subdirectory (under the base directory) -in which everything related to this builder will be placed. On the -buildmaster, this holds build status information. On the buildslave, -this is where checkouts, compiles, and tests are run. - - <br><dt><code>factory</code><dd>This is a <code>buildbot.process.factory.BuildFactory</code> instance which -controls how the build is performed. Full details appear in their own -chapter, See <a href="#Build-Process">Build Process</a>. Parameters like the location of the CVS -repository and the compile-time options used for the build are -generally provided as arguments to the factory's constructor. - - </dl> - - <p>Other optional keys may be set on each Builder: - - <dl> -<dt><code>category</code><dd>If provided, this is a string that identifies a category for the -builder to be a part of. Status clients can limit themselves to a -subset of the available categories. A common use for this is to add -new builders to your setup (for a new module, or for a new buildslave) -that do not work correctly yet and allow you to integrate them with -the active builders. You can put these new builders in a test -category, make your main status clients ignore them, and have only -private status clients pick them up. As soon as they work, you can -move them over to the active category. - - </dl> - -<div class="node"> -<p><hr> -<a name="Defining-Status-Targets"></a> -Next: <a rel="next" accesskey="n" href="#Debug-options">Debug options</a>, -Previous: <a rel="previous" accesskey="p" href="#Defining-Builders">Defining Builders</a>, -Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> - -</div> - -<h3 class="section">4.12 Defining Status Targets</h3> - -<p>The Buildmaster has a variety of ways to present build status to -various users. Each such delivery method is a “Status Target” object -in the configuration's <code>status</code> list. To add status targets, you -just append more objects to this list: - - <p><a name="index-c_005b_0027status_0027_005d-39"></a> -<pre class="example"> c['status'] = [] - - from buildbot.status import html - c['status'].append(html.Waterfall(http_port=8010)) - - from buildbot.status import mail - m = mail.MailNotifier(fromaddr="buildbot@localhost", - extraRecipients=["builds@lists.example.com"], - sendToInterestedUsers=False) - c['status'].append(m) - - from buildbot.status import words - c['status'].append(words.IRC(host="irc.example.com", nick="bb", - channels=["#example"])) -</pre> - <p>Status delivery has its own chapter, See <a href="#Status-Delivery">Status Delivery</a>, in which -all the built-in status targets are documented. - -<div class="node"> -<p><hr> -<a name="Debug-options"></a> -Previous: <a rel="previous" accesskey="p" href="#Defining-Status-Targets">Defining Status Targets</a>, -Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> - -</div> - -<h3 class="section">4.13 Debug options</h3> - -<p><a name="index-c_005b_0027debugPassword_0027_005d-40"></a>If you set <code>c['debugPassword']</code>, then you can connect to the -buildmaster with the diagnostic tool launched by <code>buildbot -debugclient MASTER:PORT</code>. From this tool, you can reload the config -file, manually force builds, and inject changes, which may be useful -for testing your buildmaster without actually commiting changes to -your repository (or before you have the Change Sources set up). The -debug tool uses the same port number as the slaves do: -<code>c['slavePortnum']</code>, and is authenticated with this password. - -<pre class="example"> c['debugPassword'] = "debugpassword" -</pre> - <p><a name="index-c_005b_0027manhole_0027_005d-41"></a>If you set <code>c['manhole']</code> to an instance of one of the classes in -<code>buildbot.manhole</code>, you can telnet or ssh into the buildmaster -and get an interactive Python shell, which may be useful for debugging -buildbot internals. It is probably only useful for buildbot -developers. It exposes full access to the buildmaster's account -(including the ability to modify and delete files), so it should not -be enabled with a weak or easily guessable password. - - <p>There are three separate <code>Manhole</code> classes. Two of them use SSH, -one uses unencrypted telnet. Two of them use a username+password -combination to grant access, one of them uses an SSH-style -<samp><span class="file">authorized_keys</span></samp> file which contains a list of ssh public keys. - - <dl> -<dt><code>manhole.AuthorizedKeysManhole</code><dd>You construct this with the name of a file that contains one SSH -public key per line, just like <samp><span class="file">~/.ssh/authorized_keys</span></samp>. If you -provide a non-absolute filename, it will be interpreted relative to -the buildmaster's base directory. - - <br><dt><code>manhole.PasswordManhole</code><dd>This one accepts SSH connections but asks for a username and password -when authenticating. It accepts only one such pair. - - <br><dt><code>manhole.TelnetManhole</code><dd>This accepts regular unencrypted telnet connections, and asks for a -username/password pair before providing access. Because this -username/password is transmitted in the clear, and because Manhole -access to the buildmaster is equivalent to granting full shell -privileges to both the buildmaster and all the buildslaves (and to all -accounts which then run code produced by the buildslaves), it is -highly recommended that you use one of the SSH manholes instead. - - </dl> - -<pre class="example"> # some examples: - from buildbot import manhole - c['manhole'] = manhole.AuthorizedKeysManhole(1234, "authorized_keys") - c['manhole'] = manhole.PasswordManhole(1234, "alice", "mysecretpassword") - c['manhole'] = manhole.TelnetManhole(1234, "bob", "snoop_my_password_please") -</pre> - <p>The <code>Manhole</code> instance can be configured to listen on a specific -port. You may wish to have this listening port bind to the loopback -interface (sometimes known as “lo0”, “localhost”, or 127.0.0.1) to -restrict access to clients which are running on the same host. - -<pre class="example"> from buildbot.manhole import PasswordManhole - c['manhole'] = PasswordManhole("tcp:9999:interface=127.0.0.1","admin","passwd") -</pre> - <p>To have the <code>Manhole</code> listen on all interfaces, use -<code>"tcp:9999"</code> or simply 9999. This port specification uses -<code>twisted.application.strports</code>, so you can make it listen on SSL -or even UNIX-domain sockets if you want. - - <p>Note that using any Manhole requires that the TwistedConch package be -installed, and that you be using Twisted version 2.0 or later. - - <p>The buildmaster's SSH server will use a different host key than the -normal sshd running on a typical unix host. This will cause the ssh -client to complain about a “host key mismatch”, because it does not -realize there are two separate servers running on the same host. To -avoid this, use a clause like the following in your <samp><span class="file">.ssh/config</span></samp> -file: - -<pre class="example"> Host remotehost-buildbot - HostName remotehost - HostKeyAlias remotehost-buildbot - Port 9999 - # use 'user' if you use PasswordManhole and your name is not 'admin'. - # if you use AuthorizedKeysManhole, this probably doesn't matter. - User admin -</pre> - <div class="node"> -<p><hr> -<a name="Getting-Source-Code-Changes"></a> -Next: <a rel="next" accesskey="n" href="#Build-Process">Build Process</a>, -Previous: <a rel="previous" accesskey="p" href="#Configuration">Configuration</a>, -Up: <a rel="up" accesskey="u" href="#Top">Top</a> - -</div> - -<h2 class="chapter">5 Getting Source Code Changes</h2> - -<p>The most common way to use the Buildbot is centered around the idea of -<code>Source Trees</code>: a directory tree filled with source code of some form -which can be compiled and/or tested. Some projects use languages that don't -involve any compilation step: nevertheless there may be a <code>build</code> phase -where files are copied or rearranged into a form that is suitable for -installation. Some projects do not have unit tests, and the Buildbot is -merely helping to make sure that the sources can compile correctly. But in -all of these cases, the thing-being-tested is a single source tree. - - <p>A Version Control System mantains a source tree, and tells the -buildmaster when it changes. The first step of each Build is typically -to acquire a copy of some version of this tree. - - <p>This chapter describes how the Buildbot learns about what Changes have -occurred. For more information on VC systems and Changes, see -<a href="#Version-Control-Systems">Version Control Systems</a>. - -<ul class="menu"> -<li><a accesskey="1" href="#Change-Sources">Change Sources</a> -<li><a accesskey="2" href="#Choosing-ChangeSources">Choosing ChangeSources</a> -<li><a accesskey="3" href="#CVSToys-_002d-PBService">CVSToys - PBService</a> -<li><a accesskey="4" href="#Mail_002dparsing-ChangeSources">Mail-parsing ChangeSources</a> -<li><a accesskey="5" href="#PBChangeSource">PBChangeSource</a> -<li><a accesskey="6" href="#P4Source">P4Source</a> -<li><a accesskey="7" href="#BonsaiPoller">BonsaiPoller</a> -<li><a accesskey="8" href="#SVNPoller">SVNPoller</a> -<li><a accesskey="9" href="#MercurialHook">MercurialHook</a> -<li><a href="#Bzr-Hook">Bzr Hook</a> -<li><a href="#Bzr-Poller">Bzr Poller</a> -</ul> - -<div class="node"> -<p><hr> -<a name="Change-Sources"></a> -Next: <a rel="next" accesskey="n" href="#Choosing-ChangeSources">Choosing ChangeSources</a>, -Previous: <a rel="previous" accesskey="p" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>, -Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> - -</div> - -<h3 class="section">5.1 Change Sources</h3> - -<!-- TODO: rework this, the one-buildmaster-one-tree thing isn't quite --> -<!-- so narrow-minded anymore --> -<p>Each Buildmaster watches a single source tree. Changes can be provided -by a variety of ChangeSource types, however any given project will -typically have only a single ChangeSource active. This section -provides a description of all available ChangeSource types and -explains how to set up each of them. - - <p>There are a variety of ChangeSources available, some of which are -meant to be used in conjunction with other tools to deliver Change -events from the VC repository to the buildmaster. - - <ul> -<li>CVSToys -This ChangeSource opens a TCP connection from the buildmaster to a -waiting FreshCVS daemon that lives on the repository machine, and -subscribes to hear about Changes. - - <li>MaildirSource -This one watches a local maildir-format inbox for email sent out by -the repository when a change is made. When a message arrives, it is -parsed to create the Change object. A variety of parsing functions are -available to accomodate different email-sending tools. - - <li>PBChangeSource -This ChangeSource listens on a local TCP socket for inbound -connections from a separate tool. Usually, this tool would be run on -the VC repository machine in a commit hook. It is expected to connect -to the TCP socket and send a Change message over the network -connection. The <samp><span class="command">buildbot sendchange</span></samp> command is one example -of a tool that knows how to send these messages, so you can write a -commit script for your VC system that calls it to deliver the Change. -There are other tools in the contrib/ directory that use the same -protocol. - - </ul> - - <p>As a quick guide, here is a list of VC systems and the ChangeSources -that might be useful with them. All of these ChangeSources are in the -<code>buildbot.changes</code> module. - - <dl> -<dt><code>CVS</code><dd> - <ul> -<li>freshcvs.FreshCVSSource (connected via TCP to the freshcvs daemon) -<li>mail.FCMaildirSource (watching for email sent by a freshcvs daemon) -<li>mail.BonsaiMaildirSource (watching for email sent by Bonsai) -<li>mail.SyncmailMaildirSource (watching for email sent by syncmail) -<li>pb.PBChangeSource (listening for connections from <code>buildbot -sendchange</code> run in a loginfo script) -<li>pb.PBChangeSource (listening for connections from a long-running -<code>contrib/viewcvspoll.py</code> polling process which examines the ViewCVS -database directly -</ul> - - <br><dt><code>SVN</code><dd> - <ul> -<li>pb.PBChangeSource (listening for connections from -<code>contrib/svn_buildbot.py</code> run in a postcommit script) -<li>pb.PBChangeSource (listening for connections from a long-running -<code>contrib/svn_watcher.py</code> or <code>contrib/svnpoller.py</code> polling -process -<li>mail.SVNCommitEmailMaildirSource (watching for email sent by commit-email.pl) -<li>svnpoller.SVNPoller (polling the SVN repository) -</ul> - - <br><dt><code>Darcs</code><dd> - <ul> -<li>pb.PBChangeSource (listening for connections from -<code>contrib/darcs_buildbot.py</code> in a commit script -</ul> - - <br><dt><code>Mercurial</code><dd> - <ul> -<li>pb.PBChangeSource (listening for connections from -<code>contrib/hg_buildbot.py</code> run in an 'incoming' hook) -<li>pb.PBChangeSource (listening for connections from -<code>buildbot/changes/hgbuildbot.py</code> run as an in-process 'changegroup' -hook) -</ul> - - <br><dt><code>Arch/Bazaar</code><dd> - <ul> -<li>pb.PBChangeSource (listening for connections from -<code>contrib/arch_buildbot.py</code> run in a commit hook) -</ul> - - <br><dt><code>Bzr (the newer Bazaar)</code><dd> - <ul> -<li>pb.PBChangeSource (listening for connections from -<code>contrib/bzr_buildbot.py</code> run in a post-change-branch-tip or commit hook) -<li><code>contrib/bzr_buildbot.py</code>'s BzrPoller (polling the Bzr repository) -</ul> - - <br><dt><code>Git</code><dd> - <ul> -<li>pb.PBChangeSource (listening for connections from -<code>contrib/git_buildbot.py</code> run in the post-receive hook) -</ul> - - </dl> - - <p>All VC systems can be driven by a PBChangeSource and the -<code>buildbot sendchange</code> tool run from some form of commit script. -If you write an email parsing function, they can also all be driven by -a suitable <code>MaildirSource</code>. - -<div class="node"> -<p><hr> -<a name="Choosing-ChangeSources"></a> -Next: <a rel="next" accesskey="n" href="#CVSToys-_002d-PBService">CVSToys - PBService</a>, -Previous: <a rel="previous" accesskey="p" href="#Change-Sources">Change Sources</a>, -Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> - -</div> - -<h3 class="section">5.2 Choosing ChangeSources</h3> - -<p>The <code>master.cfg</code> configuration file has a dictionary key named -<code>BuildmasterConfig['change_source']</code>, which holds the active -<code>IChangeSource</code> object. The config file will typically create an -object from one of the classes described below and stuff it into this -key. - - <p>Each buildmaster typically has just a single ChangeSource, since it is -only watching a single source tree. But if, for some reason, you need -multiple sources, just set <code>c['change_source']</code> to a list of -ChangeSources.. it will accept that too. - -<pre class="example"> s = FreshCVSSourceNewcred(host="host", port=4519, - user="alice", passwd="secret", - prefix="Twisted") - BuildmasterConfig['change_source'] = [s] -</pre> - <p>Each source tree has a nominal <code>top</code>. Each Change has a list of -filenames, which are all relative to this top location. The -ChangeSource is responsible for doing whatever is necessary to -accomplish this. Most sources have a <code>prefix</code> argument: a partial -pathname which is stripped from the front of all filenames provided to -that <code>ChangeSource</code>. Files which are outside this sub-tree are -ignored by the changesource: it does not generate Changes for those -files. - -<div class="node"> -<p><hr> -<a name="CVSToys---PBService"></a> -<a name="CVSToys-_002d-PBService"></a> -Next: <a rel="next" accesskey="n" href="#Mail_002dparsing-ChangeSources">Mail-parsing ChangeSources</a>, -Previous: <a rel="previous" accesskey="p" href="#Choosing-ChangeSources">Choosing ChangeSources</a>, -Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> - -</div> - -<h3 class="section">5.3 CVSToys - PBService</h3> - -<p><a name="index-buildbot_002echanges_002efreshcvs_002eFreshCVSSource-42"></a> -The <a href="http://purl.net/net/CVSToys">CVSToys</a> package provides a -server which runs on the machine that hosts the CVS repository it -watches. It has a variety of ways to distribute commit notifications, -and offers a flexible regexp-based way to filter out uninteresting -changes. One of the notification options is named <code>PBService</code> and -works by listening on a TCP port for clients. These clients subscribe -to hear about commit notifications. - - <p>The buildmaster has a CVSToys-compatible <code>PBService</code> client built -in. There are two versions of it, one for old versions of CVSToys -(1.0.9 and earlier) which used the <code>oldcred</code> authentication -framework, and one for newer versions (1.0.10 and later) which use -<code>newcred</code>. Both are classes in the -<code>buildbot.changes.freshcvs</code> package. - - <p><code>FreshCVSSourceNewcred</code> objects are created with the following -parameters: - - <dl> -<dt>‘<samp><code>host</code><span class="samp"> and </span><code>port</code></samp>’<dd>these specify where the CVSToys server can be reached - - <br><dt>‘<samp><code>user</code><span class="samp"> and </span><code>passwd</code></samp>’<dd>these specify the login information for the CVSToys server -(<code>freshcvs</code>). These must match the server's values, which are -defined in the <code>freshCfg</code> configuration file (which lives in the -CVSROOT directory of the repository). - - <br><dt>‘<samp><code>prefix</code></samp>’<dd>this is the prefix to be found and stripped from filenames delivered -by the CVSToys server. Most projects live in sub-directories of the -main repository, as siblings of the CVSROOT sub-directory, so -typically this prefix is set to that top sub-directory name. - - </dl> - -<h3 class="heading">Example</h3> - -<p>To set up the freshCVS server, add a statement like the following to -your <samp><span class="file">freshCfg</span></samp> file: - -<pre class="example"> pb = ConfigurationSet([ - (None, None, None, PBService(userpass=('foo', 'bar'), port=4519)), - ]) -</pre> - <p>This will announce all changes to a client which connects to port 4519 -using a username of 'foo' and a password of 'bar'. - - <p>Then add a clause like this to your buildmaster's <samp><span class="file">master.cfg</span></samp>: - -<pre class="example"> BuildmasterConfig['change_source'] = FreshCVSSource("cvs.example.com", 4519, - "foo", "bar", - prefix="glib/") -</pre> - <p>where "cvs.example.com" is the host that is running the FreshCVS daemon, and -"glib" is the top-level directory (relative to the repository's root) where -all your source code lives. Most projects keep one or more projects in the -same repository (along with CVSROOT/ to hold admin files like loginfo and -freshCfg); the prefix= argument tells the buildmaster to ignore everything -outside that directory, and to strip that common prefix from all pathnames -it handles. - -<div class="node"> -<p><hr> -<a name="Mail-parsing-ChangeSources"></a> -<a name="Mail_002dparsing-ChangeSources"></a> -Next: <a rel="next" accesskey="n" href="#PBChangeSource">PBChangeSource</a>, -Previous: <a rel="previous" accesskey="p" href="#CVSToys-_002d-PBService">CVSToys - PBService</a>, -Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> - -</div> - -<h3 class="section">5.4 Mail-parsing ChangeSources</h3> - -<p>Many projects publish information about changes to their source tree -by sending an email message out to a mailing list, frequently named -PROJECT-commits or PROJECT-changes. Each message usually contains a -description of the change (who made the change, which files were -affected) and sometimes a copy of the diff. Humans can subscribe to -this list to stay informed about what's happening to the source tree. - - <p>The Buildbot can also be subscribed to a -commits mailing list, and -can trigger builds in response to Changes that it hears about. The -buildmaster admin needs to arrange for these email messages to arrive -in a place where the buildmaster can find them, and configure the -buildmaster to parse the messages correctly. Once that is in place, -the email parser will create Change objects and deliver them to the -Schedulers (see see <a href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a>) just -like any other ChangeSource. - - <p>There are two components to setting up an email-based ChangeSource. -The first is to route the email messages to the buildmaster, which is -done by dropping them into a “maildir”. The second is to actually -parse the messages, which is highly dependent upon the tool that was -used to create them. Each VC system has a collection of favorite -change-emailing tools, and each has a slightly different format, so -each has a different parsing function. There is a separate -ChangeSource variant for each parsing function. - - <p>Once you've chosen a maildir location and a parsing function, create -the change source and put it in <code>c['change_source']</code>: - -<pre class="example"> from buildbot.changes.mail import SyncmailMaildirSource - c['change_source'] = SyncmailMaildirSource("~/maildir-buildbot", - prefix="/trunk/") -</pre> - <ul class="menu"> -<li><a accesskey="1" href="#Subscribing-the-Buildmaster">Subscribing the Buildmaster</a> -<li><a accesskey="2" href="#Using-Maildirs">Using Maildirs</a> -<li><a accesskey="3" href="#Parsing-Email-Change-Messages">Parsing Email Change Messages</a> -</ul> - -<div class="node"> -<p><hr> -<a name="Subscribing-the-Buildmaster"></a> -Next: <a rel="next" accesskey="n" href="#Using-Maildirs">Using Maildirs</a>, -Previous: <a rel="previous" accesskey="p" href="#Mail_002dparsing-ChangeSources">Mail-parsing ChangeSources</a>, -Up: <a rel="up" accesskey="u" href="#Mail_002dparsing-ChangeSources">Mail-parsing ChangeSources</a> - -</div> - -<h4 class="subsection">5.4.1 Subscribing the Buildmaster</h4> - -<p>The recommended way to install the buildbot is to create a dedicated -account for the buildmaster. If you do this, the account will probably -have a distinct email address (perhaps -<a href="mailto:buildmaster@example.org">buildmaster@example.org</a>). Then just arrange for this -account's email to be delivered to a suitable maildir (described in -the next section). - - <p>If the buildbot does not have its own account, “extension addresses” -can be used to distinguish between email intended for the buildmaster -and email intended for the rest of the account. In most modern MTAs, -the e.g. <code>foo@example.org</code> account has control over every email -address at example.org which begins with "foo", such that email -addressed to <a href="mailto:account-foo@example.org">account-foo@example.org</a> can be delivered to a -different destination than <a href="mailto:account-bar@example.org">account-bar@example.org</a>. qmail -does this by using separate .qmail files for the two destinations -(<samp><span class="file">.qmail-foo</span></samp> and <samp><span class="file">.qmail-bar</span></samp>, with <samp><span class="file">.qmail</span></samp> -controlling the base address and <samp><span class="file">.qmail-default</span></samp> controlling all -other extensions). Other MTAs have similar mechanisms. - - <p>Thus you can assign an extension address like -<a href="mailto:foo-buildmaster@example.org">foo-buildmaster@example.org</a> to the buildmaster, and retain -<a href="mailto:foo@example.org">foo@example.org</a> for your own use. - -<div class="node"> -<p><hr> -<a name="Using-Maildirs"></a> -Next: <a rel="next" accesskey="n" href="#Parsing-Email-Change-Messages">Parsing Email Change Messages</a>, -Previous: <a rel="previous" accesskey="p" href="#Subscribing-the-Buildmaster">Subscribing the Buildmaster</a>, -Up: <a rel="up" accesskey="u" href="#Mail_002dparsing-ChangeSources">Mail-parsing ChangeSources</a> - -</div> - -<h4 class="subsection">5.4.2 Using Maildirs</h4> - -<p>A “maildir” is a simple directory structure originally developed for -qmail that allows safe atomic update without locking. Create a base -directory with three subdirectories: “new”, “tmp”, and “cur”. -When messages arrive, they are put into a uniquely-named file (using -pids, timestamps, and random numbers) in “tmp”. When the file is -complete, it is atomically renamed into “new”. Eventually the -buildmaster notices the file in “new”, reads and parses the -contents, then moves it into “cur”. A cronjob can be used to delete -files in “cur” at leisure. - - <p>Maildirs are frequently created with the <samp><span class="command">maildirmake</span></samp> tool, -but a simple <samp><span class="command">mkdir -p ~/MAILDIR/{cur,new,tmp}</span></samp> is pretty much -equivalent. - - <p>Many modern MTAs can deliver directly to maildirs. The usual .forward -or .procmailrc syntax is to name the base directory with a trailing -slash, so something like <code>~/MAILDIR/</code> . qmail and postfix are -maildir-capable MTAs, and procmail is a maildir-capable MDA (Mail -Delivery Agent). - - <p>For MTAs which cannot put files into maildirs directly, the -“safecat” tool can be executed from a .forward file to accomplish -the same thing. - - <p>The Buildmaster uses the linux DNotify facility to receive immediate -notification when the maildir's “new” directory has changed. When -this facility is not available, it polls the directory for new -messages, every 10 seconds by default. - -<div class="node"> -<p><hr> -<a name="Parsing-Email-Change-Messages"></a> -Previous: <a rel="previous" accesskey="p" href="#Using-Maildirs">Using Maildirs</a>, -Up: <a rel="up" accesskey="u" href="#Mail_002dparsing-ChangeSources">Mail-parsing ChangeSources</a> - -</div> - -<h4 class="subsection">5.4.3 Parsing Email Change Messages</h4> - -<p>The second component to setting up an email-based ChangeSource is to -parse the actual notices. This is highly dependent upon the VC system -and commit script in use. - - <p>A couple of common tools used to create these change emails are: - - <dl> -<dt>‘<samp><span class="samp">CVS</span></samp>’<dd> - <dl> -<dt>‘<samp><span class="samp">CVSToys MailNotifier</span></samp>’<dd><a href="#FCMaildirSource">FCMaildirSource</a> -<br><dt>‘<samp><span class="samp">Bonsai notification</span></samp>’<dd><a href="#BonsaiMaildirSource">BonsaiMaildirSource</a> -<br><dt>‘<samp><span class="samp">syncmail</span></samp>’<dd><a href="#SyncmailMaildirSource">SyncmailMaildirSource</a> -</dl> - - <br><dt>‘<samp><span class="samp">SVN</span></samp>’<dd> - <dl> -<dt>‘<samp><span class="samp">svnmailer</span></samp>’<dd>http://opensource.perlig.de/en/svnmailer/ -<br><dt>‘<samp><span class="samp">commit-email.pl</span></samp>’<dd><a href="#SVNCommitEmailMaildirSource">SVNCommitEmailMaildirSource</a> -</dl> - - <br><dt>‘<samp><span class="samp">Mercurial</span></samp>’<dd> - <dl> -<dt>‘<samp><span class="samp">NotifyExtension</span></samp>’<dd>http://www.selenic.com/mercurial/wiki/index.cgi/NotifyExtension -</dl> - - <br><dt>‘<samp><span class="samp">Git</span></samp>’<dd> - <dl> -<dt>‘<samp><span class="samp">post-receive-email</span></samp>’<dd>http://git.kernel.org/?p=git/git.git;a=blob;f=contrib/hooks/post-receive-email;hb=HEAD -</dl> - - </dl> - - <p>The following sections describe the parsers available for each of -these tools. - - <p>Most of these parsers accept a <code>prefix=</code> argument, which is used -to limit the set of files that the buildmaster pays attention to. This -is most useful for systems like CVS and SVN which put multiple -projects in a single repository (or use repository names to indicate -branches). Each filename that appears in the email is tested against -the prefix: if the filename does not start with the prefix, the file -is ignored. If the filename <em>does</em> start with the prefix, that -prefix is stripped from the filename before any further processing is -done. Thus the prefix usually ends with a slash. - -<ul class="menu"> -<li><a accesskey="1" href="#FCMaildirSource">FCMaildirSource</a> -<li><a accesskey="2" href="#SyncmailMaildirSource">SyncmailMaildirSource</a> -<li><a accesskey="3" href="#BonsaiMaildirSource">BonsaiMaildirSource</a> -<li><a accesskey="4" href="#SVNCommitEmailMaildirSource">SVNCommitEmailMaildirSource</a> -</ul> - -<div class="node"> -<p><hr> -<a name="FCMaildirSource"></a> -Next: <a rel="next" accesskey="n" href="#SyncmailMaildirSource">SyncmailMaildirSource</a>, -Previous: <a rel="previous" accesskey="p" href="#Parsing-Email-Change-Messages">Parsing Email Change Messages</a>, -Up: <a rel="up" accesskey="u" href="#Parsing-Email-Change-Messages">Parsing Email Change Messages</a> - -</div> - -<h5 class="subsubsection">5.4.3.1 FCMaildirSource</h5> - -<p><a name="index-buildbot_002echanges_002email_002eFCMaildirSource-43"></a> -http://twistedmatrix.com/users/acapnotic/wares/code/CVSToys/ - - <p>This parser works with the CVSToys <code>MailNotification</code> action, -which will send email to a list of recipients for each commit. This -tends to work better than using <code>/bin/mail</code> from within the -CVSROOT/loginfo file directly, as CVSToys will batch together all -files changed during the same CVS invocation, and can provide more -information (like creating a ViewCVS URL for each file changed). - - <p>The Buildbot's <code>FCMaildirSource</code> knows for to parse these CVSToys -messages and turn them into Change objects. It can be given two -parameters: the directory name of the maildir root, and the prefix to -strip. - -<pre class="example"> from buildbot.changes.mail import FCMaildirSource - c['change_source'] = FCMaildirSource("~/maildir-buildbot") -</pre> - <div class="node"> -<p><hr> -<a name="SyncmailMaildirSource"></a> -Next: <a rel="next" accesskey="n" href="#BonsaiMaildirSource">BonsaiMaildirSource</a>, -Previous: <a rel="previous" accesskey="p" href="#FCMaildirSource">FCMaildirSource</a>, -Up: <a rel="up" accesskey="u" href="#Parsing-Email-Change-Messages">Parsing Email Change Messages</a> - -</div> - -<h5 class="subsubsection">5.4.3.2 SyncmailMaildirSource</h5> - -<p><a name="index-buildbot_002echanges_002email_002eSyncmailMaildirSource-44"></a> -http://sourceforge.net/projects/cvs-syncmail - - <p><code>SyncmailMaildirSource</code> knows how to parse the message format used by -the CVS “syncmail” script. - -<pre class="example"> from buildbot.changes.mail import SyncmailMaildirSource - c['change_source'] = SyncmailMaildirSource("~/maildir-buildbot") -</pre> - <div class="node"> -<p><hr> -<a name="BonsaiMaildirSource"></a> -Next: <a rel="next" accesskey="n" href="#SVNCommitEmailMaildirSource">SVNCommitEmailMaildirSource</a>, -Previous: <a rel="previous" accesskey="p" href="#SyncmailMaildirSource">SyncmailMaildirSource</a>, -Up: <a rel="up" accesskey="u" href="#Parsing-Email-Change-Messages">Parsing Email Change Messages</a> - -</div> - -<h5 class="subsubsection">5.4.3.3 BonsaiMaildirSource</h5> - -<p><a name="index-buildbot_002echanges_002email_002eBonsaiMaildirSource-45"></a> -http://www.mozilla.org/bonsai.html - - <p><code>BonsaiMaildirSource</code> parses messages sent out by Bonsai, the CVS -tree-management system built by Mozilla. - -<pre class="example"> from buildbot.changes.mail import BonsaiMaildirSource - c['change_source'] = BonsaiMaildirSource("~/maildir-buildbot") -</pre> - <div class="node"> -<p><hr> -<a name="SVNCommitEmailMaildirSource"></a> -Previous: <a rel="previous" accesskey="p" href="#BonsaiMaildirSource">BonsaiMaildirSource</a>, -Up: <a rel="up" accesskey="u" href="#Parsing-Email-Change-Messages">Parsing Email Change Messages</a> - -</div> - -<h5 class="subsubsection">5.4.3.4 SVNCommitEmailMaildirSource</h5> - -<p><a name="index-buildbot_002echanges_002email_002eSVNCommitEmailMaildirSource-46"></a> -<code>SVNCommitEmailMaildirSource</code> parses message sent out by the -<code>commit-email.pl</code> script, which is included in the Subversion -distribution. - - <p>It does not currently handle branches: all of the Change objects that -it creates will be associated with the default (i.e. trunk) branch. - -<pre class="example"> from buildbot.changes.mail import SVNCommitEmailMaildirSource - c['change_source'] = SVNCommitEmailMaildirSource("~/maildir-buildbot") -</pre> - <div class="node"> -<p><hr> -<a name="PBChangeSource"></a> -Next: <a rel="next" accesskey="n" href="#P4Source">P4Source</a>, -Previous: <a rel="previous" accesskey="p" href="#Mail_002dparsing-ChangeSources">Mail-parsing ChangeSources</a>, -Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> - -</div> - -<h3 class="section">5.5 PBChangeSource</h3> - -<p><a name="index-buildbot_002echanges_002epb_002ePBChangeSource-47"></a> -The last kind of ChangeSource actually listens on a TCP port for -clients to connect and push change notices <em>into</em> the -Buildmaster. This is used by the built-in <code>buildbot sendchange</code> -notification tool, as well as the VC-specific -<samp><span class="file">contrib/svn_buildbot.py</span></samp>, <samp><span class="file">contrib/arch_buildbot.py</span></samp>, -<samp><span class="file">contrib/hg_buildbot.py</span></samp> tools, and the -<code>buildbot.changes.hgbuildbot</code> hook. These tools are run by the -repository (in a commit hook script), and connect to the buildmaster -directly each time a file is comitted. This is also useful for -creating new kinds of change sources that work on a <code>push</code> model -instead of some kind of subscription scheme, for example a script -which is run out of an email .forward file. - - <p>This ChangeSource can be configured to listen on its own TCP port, or -it can share the port that the buildmaster is already using for the -buildslaves to connect. (This is possible because the -<code>PBChangeSource</code> uses the same protocol as the buildslaves, and -they can be distinguished by the <code>username</code> attribute used when -the initial connection is established). It might be useful to have it -listen on a different port if, for example, you wanted to establish -different firewall rules for that port. You could allow only the SVN -repository machine access to the <code>PBChangeSource</code> port, while -allowing only the buildslave machines access to the slave port. Or you -could just expose one port and run everything over it. <em>Note: -this feature is not yet implemented, the PBChangeSource will always -share the slave port and will always have a </em><code>user</code><em> name of -</em><code>change</code><em>, and a passwd of </em><code>changepw</code><em>. These limitations will -be removed in the future.</em>. - - <p>The <code>PBChangeSource</code> is created with the following arguments. All -are optional. - - <dl> -<dt>‘<samp><code>port</code></samp>’<dd>which port to listen on. If <code>None</code> (which is the default), it -shares the port used for buildslave connections. <em>Not -Implemented, always set to </em><code>None</code>. - - <br><dt>‘<samp><code>user</code><span class="samp"> and </span><code>passwd</code></samp>’<dd>The user/passwd account information that the client program must use -to connect. Defaults to <code>change</code> and <code>changepw</code>. <em>Not -Implemented, </em><code>user</code><em> is currently always set to </em><code>change</code><em>, -</em><code>passwd</code><em> is always set to </em><code>changepw</code>. - - <br><dt>‘<samp><code>prefix</code></samp>’<dd>The prefix to be found and stripped from filenames delivered over the -connection. Any filenames which do not start with this prefix will be -removed. If all the filenames in a given Change are removed, the that -whole Change will be dropped. This string should probably end with a -directory separator. - - <p>This is useful for changes coming from version control systems that -represent branches as parent directories within the repository (like -SVN and Perforce). Use a prefix of 'trunk/' or -'project/branches/foobranch/' to only follow one branch and to get -correct tree-relative filenames. Without a prefix, the PBChangeSource -will probably deliver Changes with filenames like <samp><span class="file">trunk/foo.c</span></samp> -instead of just <samp><span class="file">foo.c</span></samp>. Of course this also depends upon the -tool sending the Changes in (like <samp><span class="command">buildbot sendchange</span></samp>) and -what filenames it is delivering: that tool may be filtering and -stripping prefixes at the sending end. - - </dl> - -<div class="node"> -<p><hr> -<a name="P4Source"></a> -Next: <a rel="next" accesskey="n" href="#BonsaiPoller">BonsaiPoller</a>, -Previous: <a rel="previous" accesskey="p" href="#PBChangeSource">PBChangeSource</a>, -Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> - -</div> - -<h3 class="section">5.6 P4Source</h3> - -<p><a name="index-buildbot_002echanges_002ep4poller_002eP4Source-48"></a> -The <code>P4Source</code> periodically polls a <a href="http://www.perforce.com/">Perforce</a> depot for changes. It accepts the following arguments: - - <dl> -<dt>‘<samp><code>p4base</code></samp>’<dd>The base depot path to watch, without the trailing '/...'. - - <br><dt>‘<samp><code>p4port</code></samp>’<dd>The Perforce server to connect to (as host:port). - - <br><dt>‘<samp><code>p4user</code></samp>’<dd>The Perforce user. - - <br><dt>‘<samp><code>p4passwd</code></samp>’<dd>The Perforce password. - - <br><dt>‘<samp><code>p4bin</code></samp>’<dd>An optional string parameter. Specify the location of the perforce command -line binary (p4). You only need to do this if the perforce binary is not -in the path of the buildbot user. Defaults to “p4”. - - <br><dt>‘<samp><code>split_file</code></samp>’<dd>A function that maps a pathname, without the leading <code>p4base</code>, to a -(branch, filename) tuple. The default just returns (None, branchfile), -which effectively disables branch support. You should supply a function -which understands your repository structure. - - <br><dt>‘<samp><code>pollinterval</code></samp>’<dd>How often to poll, in seconds. Defaults to 600 (10 minutes). - - <br><dt>‘<samp><code>histmax</code></samp>’<dd>The maximum number of changes to inspect at a time. If more than this -number occur since the last poll, older changes will be silently -ignored. -</dl> - -<h3 class="heading">Example</h3> - -<p>This configuration uses the <code>P4PORT</code>, <code>P4USER</code>, and <code>P4PASSWD</code> -specified in the buildmaster's environment. It watches a project in which the -branch name is simply the next path component, and the file is all path -components after. - -<pre class="example"> import buildbot.changes.p4poller - s = p4poller.P4Source(p4base='//depot/project/', - split_file=lambda branchfile: branchfile.split('/',1), - ) - c['change_source'] = s -</pre> - <div class="node"> -<p><hr> -<a name="BonsaiPoller"></a> -Next: <a rel="next" accesskey="n" href="#SVNPoller">SVNPoller</a>, -Previous: <a rel="previous" accesskey="p" href="#P4Source">P4Source</a>, -Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> - -</div> - -<h3 class="section">5.7 BonsaiPoller</h3> - -<p><a name="index-buildbot_002echanges_002ebonsaipoller_002eBonsaiPoller-49"></a> -The <code>BonsaiPoller</code> periodically polls a Bonsai server. This is a -CGI script accessed through a web server that provides information -about a CVS tree, for example the Mozilla bonsai server at -<a href="http://bonsai.mozilla.org">http://bonsai.mozilla.org</a>. Bonsai servers are usable by both -humans and machines. In this case, the buildbot's change source forms -a query which asks about any files in the specified branch which have -changed since the last query. - - <p>Please take a look at the BonsaiPoller docstring for details about the -arguments it accepts. - -<div class="node"> -<p><hr> -<a name="SVNPoller"></a> -Next: <a rel="next" accesskey="n" href="#MercurialHook">MercurialHook</a>, -Previous: <a rel="previous" accesskey="p" href="#BonsaiPoller">BonsaiPoller</a>, -Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> - -</div> - -<h3 class="section">5.8 SVNPoller</h3> - -<p><a name="index-buildbot_002echanges_002esvnpoller_002eSVNPoller-50"></a> -The <code>buildbot.changes.svnpoller.SVNPoller</code> is a ChangeSource -which periodically polls a <a href="http://subversion.tigris.org/">Subversion</a> repository for new revisions, by running the <code>svn -log</code> command in a subshell. It can watch a single branch or multiple -branches. - - <p><code>SVNPoller</code> accepts the following arguments: - - <dl> -<dt><code>svnurl</code><dd>The base URL path to watch, like -<code>svn://svn.twistedmatrix.com/svn/Twisted/trunk</code>, or -<code>http://divmod.org/svn/Divmod/</code>, or even -<code>file:///home/svn/Repository/ProjectA/branches/1.5/</code>. This must -include the access scheme, the location of the repository (both the -hostname for remote ones, and any additional directory names necessary -to get to the repository), and the sub-path within the repository's -virtual filesystem for the project and branch of interest. - - <p>The <code>SVNPoller</code> will only pay attention to files inside the -subdirectory specified by the complete svnurl. - - <br><dt><code>split_file</code><dd>A function to convert pathnames into (branch, relative_pathname) -tuples. Use this to explain your repository's branch-naming policy to -<code>SVNPoller</code>. This function must accept a single string and return -a two-entry tuple. There are a few utility functions in -<code>buildbot.changes.svnpoller</code> that can be used as a -<code>split_file</code> function, see below for details. - - <p>The default value always returns (None, path), which indicates that -all files are on the trunk. - - <p>Subclasses of <code>SVNPoller</code> can override the <code>split_file</code> -method instead of using the <code>split_file=</code> argument. - - <br><dt><code>svnuser</code><dd>An optional string parameter. If set, the <code>--user</code> argument will -be added to all <code>svn</code> commands. Use this if you have to -authenticate to the svn server before you can do <code>svn info</code> or -<code>svn log</code> commands. - - <br><dt><code>svnpasswd</code><dd>Like <code>svnuser</code>, this will cause a <code>--password</code> argument to -be passed to all svn commands. - - <br><dt><code>pollinterval</code><dd>How often to poll, in seconds. Defaults to 600 (checking once every 10 -minutes). Lower this if you want the buildbot to notice changes -faster, raise it if you want to reduce the network and CPU load on -your svn server. Please be considerate of public SVN repositories by -using a large interval when polling them. - - <br><dt><code>histmax</code><dd>The maximum number of changes to inspect at a time. Every POLLINTERVAL -seconds, the <code>SVNPoller</code> asks for the last HISTMAX changes and -looks through them for any ones it does not already know about. If -more than HISTMAX revisions have been committed since the last poll, -older changes will be silently ignored. Larger values of histmax will -cause more time and memory to be consumed on each poll attempt. -<code>histmax</code> defaults to 100. - - <br><dt><code>svnbin</code><dd>This controls the <code>svn</code> executable to use. If subversion is -installed in a weird place on your system (outside of the -buildmaster's <code>$PATH</code>), use this to tell <code>SVNPoller</code> where -to find it. The default value of “svn” will almost always be -sufficient. - - </dl> - -<h3 class="heading">Branches</h3> - -<p>Each source file that is tracked by a Subversion repository has a -fully-qualified SVN URL in the following form: -(REPOURL)(PROJECT-plus-BRANCH)(FILEPATH). When you create the -<code>SVNPoller</code>, you give it a <code>svnurl</code> value that includes all -of the REPOURL and possibly some portion of the PROJECT-plus-BRANCH -string. The <code>SVNPoller</code> is responsible for producing Changes that -contain a branch name and a FILEPATH (which is relative to the top of -a checked-out tree). The details of how these strings are split up -depend upon how your repository names its branches. - -<h4 class="subheading">PROJECT/BRANCHNAME/FILEPATH repositories</h4> - -<p>One common layout is to have all the various projects that share a -repository get a single top-level directory each. Then under a given -project's directory, you get two subdirectories, one named “trunk” -and another named “branches”. Under “branches” you have a bunch of -other directories, one per branch, with names like “1.5.x” and -“testing”. It is also common to see directories like “tags” and -“releases” next to “branches” and “trunk”. - - <p>For example, the Twisted project has a subversion server on -“svn.twistedmatrix.com” that hosts several sub-projects. The -repository is available through a SCHEME of “svn:”. The primary -sub-project is Twisted, of course, with a repository root of -“svn://svn.twistedmatrix.com/svn/Twisted”. Another sub-project is -Informant, with a root of -“svn://svn.twistedmatrix.com/svn/Informant”, etc. Inside any -checked-out Twisted tree, there is a file named bin/trial (which is -used to run unit test suites). - - <p>The trunk for Twisted is in -“svn://svn.twistedmatrix.com/svn/Twisted/trunk”, and the -fully-qualified SVN URL for the trunk version of <code>trial</code> would be -“svn://svn.twistedmatrix.com/svn/Twisted/trunk/bin/trial”. The same -SVNURL for that file on a branch named “1.5.x” would be -“svn://svn.twistedmatrix.com/svn/Twisted/branches/1.5.x/bin/trial”. - - <p>To set up a <code>SVNPoller</code> that watches the Twisted trunk (and -nothing else), we would use the following: - -<pre class="example"> from buildbot.changes.svnpoller import SVNPoller - c['change_source'] = SVNPoller("svn://svn.twistedmatrix.com/svn/Twisted/trunk") -</pre> - <p>In this case, every Change that our <code>SVNPoller</code> produces will -have <code>.branch=None</code>, to indicate that the Change is on the trunk. -No other sub-projects or branches will be tracked. - - <p>If we want our ChangeSource to follow multiple branches, we have to do -two things. First we have to change our <code>svnurl=</code> argument to -watch more than just “.../Twisted/trunk”. We will set it to -“.../Twisted” so that we'll see both the trunk and all the branches. -Second, we have to tell <code>SVNPoller</code> how to split the -(PROJECT-plus-BRANCH)(FILEPATH) strings it gets from the repository -out into (BRANCH) and (FILEPATH) pairs. - - <p>We do the latter by providing a “split_file” function. This function -is responsible for splitting something like -“branches/1.5.x/bin/trial” into <code>branch</code>=”branches/1.5.x” and -<code>filepath</code>=”bin/trial”. This function is always given a string -that names a file relative to the subdirectory pointed to by the -<code>SVNPoller</code>'s <code>svnurl=</code> argument. It is expected to return a -(BRANCHNAME, FILEPATH) tuple (in which FILEPATH is relative to the -branch indicated), or None to indicate that the file is outside any -project of interest. - - <p>(note that we want to see “branches/1.5.x” rather than just -“1.5.x” because when we perform the SVN checkout, we will probably -append the branch name to the baseURL, which requires that we keep the -“branches” component in there. Other VC schemes use a different -approach towards branches and may not require this artifact.) - - <p>If your repository uses this same PROJECT/BRANCH/FILEPATH naming -scheme, the following function will work: - -<pre class="example"> def split_file_branches(path): - pieces = path.split('/') - if pieces[0] == 'trunk': - return (None, '/'.join(pieces[1:])) - elif pieces[0] == 'branches': - return ('/'.join(pieces[0:2]), - '/'.join(pieces[2:])) - else: - return None -</pre> - <p>This function is provided as -<code>buildbot.changes.svnpoller.split_file_branches</code> for your -convenience. So to have our Twisted-watching <code>SVNPoller</code> follow -multiple branches, we would use this: - -<pre class="example"> from buildbot.changes.svnpoller import SVNPoller, split_file_branches - c['change_source'] = SVNPoller("svn://svn.twistedmatrix.com/svn/Twisted", - split_file=split_file_branches) -</pre> - <p>Changes for all sorts of branches (with names like “branches/1.5.x”, -and None to indicate the trunk) will be delivered to the Schedulers. -Each Scheduler is then free to use or ignore each branch as it sees -fit. - -<h4 class="subheading">BRANCHNAME/PROJECT/FILEPATH repositories</h4> - -<p>Another common way to organize a Subversion repository is to put the -branch name at the top, and the projects underneath. This is -especially frequent when there are a number of related sub-projects -that all get released in a group. - - <p>For example, Divmod.org hosts a project named “Nevow” as well as one -named “Quotient”. In a checked-out Nevow tree there is a directory -named “formless” that contains a python source file named -“webform.py”. This repository is accessible via webdav (and thus -uses an “http:” scheme) through the divmod.org hostname. There are -many branches in this repository, and they use a -(BRANCHNAME)/(PROJECT) naming policy. - - <p>The fully-qualified SVN URL for the trunk version of webform.py is -<code>http://divmod.org/svn/Divmod/trunk/Nevow/formless/webform.py</code>. -You can do an <code>svn co</code> with that URL and get a copy of the latest -version. The 1.5.x branch version of this file would have a URL of -<code>http://divmod.org/svn/Divmod/branches/1.5.x/Nevow/formless/webform.py</code>. -The whole Nevow trunk would be checked out with -<code>http://divmod.org/svn/Divmod/trunk/Nevow</code>, while the Quotient -trunk would be checked out using -<code>http://divmod.org/svn/Divmod/trunk/Quotient</code>. - - <p>Now suppose we want to have an <code>SVNPoller</code> that only cares about -the Nevow trunk. This case looks just like the PROJECT/BRANCH layout -described earlier: - -<pre class="example"> from buildbot.changes.svnpoller import SVNPoller - c['change_source'] = SVNPoller("http://divmod.org/svn/Divmod/trunk/Nevow") -</pre> - <p>But what happens when we want to track multiple Nevow branches? We -have to point our <code>svnurl=</code> high enough to see all those -branches, but we also don't want to include Quotient changes (since -we're only building Nevow). To accomplish this, we must rely upon the -<code>split_file</code> function to help us tell the difference between -files that belong to Nevow and those that belong to Quotient, as well -as figuring out which branch each one is on. - -<pre class="example"> from buildbot.changes.svnpoller import SVNPoller - c['change_source'] = SVNPoller("http://divmod.org/svn/Divmod", - split_file=my_file_splitter) -</pre> - <p>The <code>my_file_splitter</code> function will be called with -repository-relative pathnames like: - - <dl> -<dt><code>trunk/Nevow/formless/webform.py</code><dd>This is a Nevow file, on the trunk. We want the Change that includes this -to see a filename of <code>formless/webform.py"</code>, and a branch of None - - <br><dt><code>branches/1.5.x/Nevow/formless/webform.py</code><dd>This is a Nevow file, on a branch. We want to get -branch=”branches/1.5.x” and filename=”formless/webform.py”. - - <br><dt><code>trunk/Quotient/setup.py</code><dd>This is a Quotient file, so we want to ignore it by having -<code>my_file_splitter</code> return None. - - <br><dt><code>branches/1.5.x/Quotient/setup.py</code><dd>This is also a Quotient file, which should be ignored. -</dl> - - <p>The following definition for <code>my_file_splitter</code> will do the job: - -<pre class="example"> def my_file_splitter(path): - pieces = path.split('/') - if pieces[0] == 'trunk': - branch = None - pieces.pop(0) # remove 'trunk' - elif pieces[0] == 'branches': - pieces.pop(0) # remove 'branches' - # grab branch name - branch = 'branches/' + pieces.pop(0) - else: - return None # something weird - projectname = pieces.pop(0) - if projectname != 'Nevow': - return None # wrong project - return (branch, '/'.join(pieces)) -</pre> - <div class="node"> -<p><hr> -<a name="MercurialHook"></a> -Next: <a rel="next" accesskey="n" href="#Bzr-Hook">Bzr Hook</a>, -Previous: <a rel="previous" accesskey="p" href="#SVNPoller">SVNPoller</a>, -Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> - -</div> - -<h3 class="section">5.9 MercurialHook</h3> - -<p>Since Mercurial is written in python, the hook script can invoke -Buildbot's <code>sendchange</code> function directly, rather than having to -spawn an external process. This function delivers the same sort of -changes as <code>buildbot sendchange</code> and the various hook scripts in -contrib/, so you'll need to add a <code>pb.PBChangeSource</code> to your -buildmaster to receive these changes. - - <p>To set this up, first choose a Mercurial repository that represents -your central “official” source tree. This will be the same -repository that your buildslaves will eventually pull from. Install -Buildbot on the machine that hosts this repository, using the same -version of python as Mercurial is using (so that the Mercurial hook -can import code from buildbot). Then add the following to the -<code>.hg/hgrc</code> file in that repository, replacing the buildmaster -hostname/portnumber as appropriate for your buildbot: - -<pre class="example"> [hooks] - changegroup.buildbot = python:buildbot.changes.hgbuildbot.hook - - [hgbuildbot] - master = buildmaster.example.org:9987 -</pre> - <p>(Note that Mercurial lets you define multiple <code>changegroup</code> hooks -by giving them distinct names, like <code>changegroup.foo</code> and -<code>changegroup.bar</code>, which is why we use -<code>changegroup.buildbot</code> in this example. There is nothing magical -about the “buildbot” suffix in the hook name. The -<code>[hgbuildbot]</code> section <em>is</em> special, however, as it is the -only section that the buildbot hook pays attention to.) - - <p>Also note that this runs as a <code>changegroup</code> hook, rather than as -an <code>incoming</code> hook. The <code>changegroup</code> hook is run with -multiple revisions at a time (say, if multiple revisions are being -pushed to this repository in a single <samp><span class="command">hg push</span></samp> command), -whereas the <code>incoming</code> hook is run with just one revision at a -time. The <code>hgbuildbot.hook</code> function will only work with the -<code>changegroup</code> hook. - - <p>The <code>[hgbuildbot]</code> section has two other parameters that you -might specify, both of which control the name of the branch that is -attached to the changes coming from this hook. - - <p>One common branch naming policy for Mercurial repositories is to use -it just like Darcs: each branch goes into a separate repository, and -all the branches for a single project share a common parent directory. -For example, you might have <samp><span class="file">/var/repos/PROJECT/trunk/</span></samp> and -<samp><span class="file">/var/repos/PROJECT/release</span></samp>. To use this style, use the -<code>branchtype = dirname</code> setting, which simply uses the last -component of the repository's enclosing directory as the branch name: - -<pre class="example"> [hgbuildbot] - master = buildmaster.example.org:9987 - branchtype = dirname -</pre> - <p>Another approach is to use Mercurial's built-in branches (the kind -created with <samp><span class="command">hg branch</span></samp> and listed with <samp><span class="command">hg -branches</span></samp>). This feature associates persistent names with particular -lines of descent within a single repository. (note that the buildbot -<code>source.Mercurial</code> checkout step does not yet support this kind -of branch). To have the commit hook deliver this sort of branch name -with the Change object, use <code>branchtype = inrepo</code>: - -<pre class="example"> [hgbuildbot] - master = buildmaster.example.org:9987 - branchtype = inrepo -</pre> - <p>Finally, if you want to simply specify the branchname directly, for -all changes, use <code>branch = BRANCHNAME</code>. This overrides -<code>branchtype</code>: - -<pre class="example"> [hgbuildbot] - master = buildmaster.example.org:9987 - branch = trunk -</pre> - <p>If you use <code>branch=</code> like this, you'll need to put a separate -.hgrc in each repository. If you use <code>branchtype=</code>, you may be -able to use the same .hgrc for all your repositories, stored in -<samp><span class="file">~/.hgrc</span></samp> or <samp><span class="file">/etc/mercurial/hgrc</span></samp>. - -<div class="node"> -<p><hr> -<a name="Bzr-Hook"></a> -Next: <a rel="next" accesskey="n" href="#Bzr-Poller">Bzr Poller</a>, -Previous: <a rel="previous" accesskey="p" href="#MercurialHook">MercurialHook</a>, -Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> - -</div> - -<h3 class="section">5.10 Bzr Hook</h3> - -<p>Bzr is also written in Python, and the Bzr hook depends on Twisted to send the -changes. - - <p>To install, put <code>contrib/bzr_buildbot.py</code> in one of your plugins -locations a bzr plugins directory (e.g., -<code>~/.bazaar/plugins</code>). Then, in one of your bazaar conf files (e.g., -<code>~/.bazaar/locations.conf</code>), set the location you want to connect with buildbot -with these keys: - - <dl> -<dt><code>buildbot_on</code><dd>one of 'commit', 'push, or 'change'. Turns the plugin on to report changes via -commit, changes via push, or any changes to the trunk. 'change' is -recommended. - - <br><dt><code>buildbot_server</code><dd>(required to send to a buildbot master) the URL of the buildbot master to -which you will connect (as of this writing, the same server and port to which -slaves connect). - - <br><dt><code>buildbot_port</code><dd>(optional, defaults to 9989) the port of the buildbot master to which you will -connect (as of this writing, the same server and port to which slaves connect) - - <br><dt><code>buildbot_pqm</code><dd>(optional, defaults to not pqm) Normally, the user that commits the revision -is the user that is responsible for the change. When run in a pqm (Patch Queue -Manager, see https://launchpad.net/pqm) environment, the user that commits is -the Patch Queue Manager, and the user that committed the *parent* revision is -responsible for the change. To turn on the pqm mode, set this value to any of -(case-insensitive) "Yes", "Y", "True", or "T". - - <br><dt><code>buildbot_dry_run</code><dd>(optional, defaults to not a dry run) Normally, the post-commit hook will -attempt to communicate with the configured buildbot server and port. If this -parameter is included and any of (case-insensitive) "Yes", "Y", "True", or -"T", then the hook will simply print what it would have sent, but not attempt -to contact the buildbot master. - - <br><dt><code>buildbot_send_branch_name</code><dd>(optional, defaults to not sending the branch name) If your buildbot's bzr -source build step uses a repourl, do *not* turn this on. If your buildbot's -bzr build step uses a baseURL, then you may set this value to any of -(case-insensitive) "Yes", "Y", "True", or "T" to have the buildbot master -append the branch name to the baseURL. - - </dl> - - <p>When buildbot no longer has a hardcoded password, it will be a configuration -option here as well. - - <p>Here's a simple example that you might have in your -<code>~/.bazaar/locations.conf</code>. - -<pre class="example"> [chroot-*:///var/local/myrepo/mybranch] - buildbot_on = change - buildbot_server = localhost -</pre> - <div class="node"> -<p><hr> -<a name="Bzr-Poller"></a> -Previous: <a rel="previous" accesskey="p" href="#Bzr-Hook">Bzr Hook</a>, -Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> - -</div> - -<h3 class="section">5.11 Bzr Poller</h3> - -<p>If you cannot insert a Bzr hook in the server, you can use the Bzr Poller. To -use, put <code>contrib/bzr_buildbot.py</code> somewhere that your buildbot -configuration can import it. Even putting it in the same directory as the master.cfg -should work. Install the poller in the buildbot configuration as with any -other change source. Minimally, provide a URL that you want to poll (bzr://, -bzr+ssh://, or lp:), though make sure the buildbot user has necessary -privileges. You may also want to specify these optional values. - - <dl> -<dt><code>poll_interval</code><dd>The number of seconds to wait between polls. Defaults to 10 minutes. - - <br><dt><code>branch_name</code><dd>Any value to be used as the branch name. Defaults to None, or specify a -string, or specify the constants from <code>bzr_buildbot.py</code> SHORT or FULL to -get the short branch name or full branch address. - - <br><dt><code>blame_merge_author</code><dd>normally, the user that commits the revision is the user that is responsible -for the change. When run in a pqm (Patch Queue Manager, see -https://launchpad.net/pqm) environment, the user that commits is the Patch -Queue Manager, and the user that committed the merged, *parent* revision is -responsible for the change. set this value to True if this is pointed against -a PQM-managed branch. -</dl> - -<div class="node"> -<p><hr> -<a name="Build-Process"></a> -Next: <a rel="next" accesskey="n" href="#Status-Delivery">Status Delivery</a>, -Previous: <a rel="previous" accesskey="p" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>, -Up: <a rel="up" accesskey="u" href="#Top">Top</a> - -</div> - -<h2 class="chapter">6 Build Process</h2> - -<p>A <code>Build</code> object is responsible for actually performing a build. -It gets access to a remote <code>SlaveBuilder</code> where it may run -commands, and a <code>BuildStatus</code> object where it must emit status -events. The <code>Build</code> is created by the Builder's -<code>BuildFactory</code>. - - <p>The default <code>Build</code> class is made up of a fixed sequence of -<code>BuildSteps</code>, executed one after another until all are complete -(or one of them indicates that the build should be halted early). The -default <code>BuildFactory</code> creates instances of this <code>Build</code> -class with a list of <code>BuildSteps</code>, so the basic way to configure -the build is to provide a list of <code>BuildSteps</code> to your -<code>BuildFactory</code>. - - <p>More complicated <code>Build</code> subclasses can make other decisions: -execute some steps only if certain files were changed, or if certain -previous steps passed or failed. The base class has been written to -allow users to express basic control flow without writing code, but -you can always subclass and customize to achieve more specialized -behavior. - -<ul class="menu"> -<li><a accesskey="1" href="#Build-Steps">Build Steps</a> -<li><a accesskey="2" href="#Interlocks">Interlocks</a> -<li><a accesskey="3" href="#Build-Factories">Build Factories</a> -</ul> - -<div class="node"> -<p><hr> -<a name="Build-Steps"></a> -Next: <a rel="next" accesskey="n" href="#Interlocks">Interlocks</a>, -Previous: <a rel="previous" accesskey="p" href="#Build-Process">Build Process</a>, -Up: <a rel="up" accesskey="u" href="#Build-Process">Build Process</a> - -</div> - -<h3 class="section">6.1 Build Steps</h3> - -<p><code>BuildStep</code>s are usually specified in the buildmaster's -configuration file, in a list that goes into the <code>BuildFactory</code>. -The <code>BuildStep</code> instances in this list are used as templates to -construct new independent copies for each build (so that state can be -kept on the <code>BuildStep</code> in one build without affecting a later -build). Each <code>BuildFactory</code> can be created with a list of steps, -or the factory can be created empty and then steps added to it using -the <code>addStep</code> method: - -<pre class="example"> from buildbot.steps import source, shell - from buildbot.process import factory - - f = factory.BuildFactory() - f.addStep(source.SVN(svnurl="http://svn.example.org/Trunk/")) - f.addStep(shell.ShellCommand(command=["make", "all"])) - f.addStep(shell.ShellCommand(command=["make", "test"])) -</pre> - <p>In earlier versions (0.7.5 and older), these steps were specified with -a tuple of (step_class, keyword_arguments). Steps can still be -specified this way, but the preferred form is to pass actual -<code>BuildStep</code> instances to <code>addStep</code>, because that gives the -<code>BuildStep</code> class a chance to do some validation on the -arguments. - - <p>If you have a common set of steps which are used in several factories, the -<code>addSteps</code> method may be handy. It takes an iterable of <code>BuildStep</code> -instances. - -<pre class="example"> setup_steps = [ - source.SVN(svnurl="http://svn.example.org/Trunk/") - shell.ShellCommand(command="./setup") - ] - quick = factory.BuildFactory() - quick.addSteps(setup_steps) - quick.addStep(shell.shellCommand(command="make quick")) -</pre> - <p>The rest of this section lists all the standard BuildStep objects -available for use in a Build, and the parameters which can be used to -control each. - -<ul class="menu"> -<li><a accesskey="1" href="#Common-Parameters">Common Parameters</a> -<li><a accesskey="2" href="#Using-Build-Properties">Using Build Properties</a> -<li><a accesskey="3" href="#Source-Checkout">Source Checkout</a> -<li><a accesskey="4" href="#ShellCommand">ShellCommand</a> -<li><a accesskey="5" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a> -<li><a accesskey="6" href="#Python-BuildSteps">Python BuildSteps</a> -<li><a accesskey="7" href="#Transferring-Files">Transferring Files</a> -<li><a accesskey="8" href="#Steps-That-Run-on-the-Master">Steps That Run on the Master</a> -<li><a accesskey="9" href="#Triggering-Schedulers">Triggering Schedulers</a> -<li><a href="#Writing-New-BuildSteps">Writing New BuildSteps</a> -</ul> - -<div class="node"> -<p><hr> -<a name="Common-Parameters"></a> -Next: <a rel="next" accesskey="n" href="#Using-Build-Properties">Using Build Properties</a>, -Previous: <a rel="previous" accesskey="p" href="#Build-Steps">Build Steps</a>, -Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> - -</div> - -<h4 class="subsection">6.1.1 Common Parameters</h4> - -<p>The standard <code>Build</code> runs a series of <code>BuildStep</code>s in order, -only stopping when it runs out of steps or if one of them requests -that the build be halted. It collects status information from each one -to create an overall build status (of SUCCESS, WARNINGS, or FAILURE). - - <p>All BuildSteps accept some common parameters. Some of these control -how their individual status affects the overall build. Others are used -to specify which <code>Locks</code> (see see <a href="#Interlocks">Interlocks</a>) should be -acquired before allowing the step to run. - - <p>Arguments common to all <code>BuildStep</code> subclasses: - - <dl> -<dt><code>name</code><dd>the name used to describe the step on the status display. It is also -used to give a name to any LogFiles created by this step. - - <br><dt><code>haltOnFailure</code><dd>if True, a FAILURE of this build step will cause the build to halt -immediately. Steps with <code>alwaysRun=True</code> are still run. Generally -speaking, haltOnFailure implies flunkOnFailure (the default for most -BuildSteps). In some cases, particularly series of tests, it makes sense -to haltOnFailure if something fails early on but not flunkOnFailure. -This can be achieved with haltOnFailure=True, flunkOnFailure=False. - - <br><dt><code>flunkOnWarnings</code><dd>when True, a WARNINGS or FAILURE of this build step will mark the -overall build as FAILURE. The remaining steps will still be executed. - - <br><dt><code>flunkOnFailure</code><dd>when True, a FAILURE of this build step will mark the overall build as -a FAILURE. The remaining steps will still be executed. - - <br><dt><code>warnOnWarnings</code><dd>when True, a WARNINGS or FAILURE of this build step will mark the -overall build as having WARNINGS. The remaining steps will still be -executed. - - <br><dt><code>warnOnFailure</code><dd>when True, a FAILURE of this build step will mark the overall build as -having WARNINGS. The remaining steps will still be executed. - - <br><dt><code>alwaysRun</code><dd>if True, this build step will always be run, even if a previous buildstep -with <code>haltOnFailure=True</code> has failed. - - <br><dt><code>locks</code><dd>a list of Locks (instances of <code>buildbot.locks.SlaveLock</code> or -<code>buildbot.locks.MasterLock</code>) that should be acquired before -starting this Step. The Locks will be released when the step is -complete. Note that this is a list of actual Lock instances, not -names. Also note that all Locks must have unique names. - - </dl> - -<div class="node"> -<p><hr> -<a name="Using-Build-Properties"></a> -Next: <a rel="next" accesskey="n" href="#Source-Checkout">Source Checkout</a>, -Previous: <a rel="previous" accesskey="p" href="#Common-Parameters">Common Parameters</a>, -Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> - -</div> - -<h4 class="subsection">6.1.2 Using Build Properties</h4> - -<p><a name="index-Properties-51"></a> -Build properties are a generalized way to provide configuration -information to build steps; see <a href="#Build-Properties">Build Properties</a>. - - <p>Some build properties are inherited from external sources – global -properties, schedulers, or buildslaves. Some build properties are -set when the build starts, such as the SourceStamp information. Other -properties can be set by BuildSteps as they run, for example the -various Source steps will set the <code>got_revision</code> property to the -source revision that was actually checked out (which can be useful -when the SourceStamp in use merely requested the “latest revision”: -<code>got_revision</code> will tell you what was actually built). - - <p>In custom BuildSteps, you can get and set the build properties with -the <code>getProperty</code>/<code>setProperty</code> methods. Each takes a string -for the name of the property, and returns or accepts an -arbitrary<a rel="footnote" href="#fn-7" name="fnd-7"><sup>7</sup></a> object. For example: - -<pre class="example"> class MakeTarball(ShellCommand): - def start(self): - if self.getProperty("os") == "win": - self.setCommand([ ... ]) # windows-only command - else: - self.setCommand([ ... ]) # equivalent for other systems - ShellCommand.start(self) -</pre> - <h3 class="heading">WithProperties</h3> - -<p><a name="index-WithProperties-52"></a> -You can use build properties in ShellCommands by using the -<code>WithProperties</code> wrapper when setting the arguments of -the ShellCommand. This interpolates the named build properties -into the generated shell command. Most step parameters accept -<code>WithProperties</code>. Please file bugs for any parameters which -do not. - -<pre class="example"> from buildbot.steps.shell import ShellCommand - from buildbot.process.properties import WithProperties - - f.addStep(ShellCommand( - command=["tar", "czf", - WithProperties("build-%s.tar.gz", "revision"), - "source"])) -</pre> - <p>If this BuildStep were used in a tree obtained from Subversion, it -would create a tarball with a name like <samp><span class="file">build-1234.tar.gz</span></samp>. - - <p>The <code>WithProperties</code> function does <code>printf</code>-style string -interpolation, using strings obtained by calling -<code>build.getProperty(propname)</code>. Note that for every <code>%s</code> (or -<code>%d</code>, etc), you must have exactly one additional argument to -indicate which build property you want to insert. - - <p>You can also use python dictionary-style string interpolation by using -the <code>%(propname)s</code> syntax. In this form, the property name goes -in the parentheses, and WithProperties takes <em>no</em> additional -arguments: - -<pre class="example"> f.addStep(ShellCommand( - command=["tar", "czf", - WithProperties("build-%(revision)s.tar.gz"), - "source"])) -</pre> - <p>Don't forget the extra “s” after the closing parenthesis! This is -the cause of many confusing errors. - - <p>The dictionary-style interpolation supports a number of more advanced -syntaxes, too. - - <dl> -<dt><code>propname:-replacement</code><dd>If <code>propname</code> exists, substitute its value; otherwise, -substitute <code>replacement</code>. <code>replacement</code> may be empty -(<code>%(propname:-)s</code>) - - <br><dt><code>propname:+replacement</code><dd>If <code>propname</code> exists, substitute <code>replacement</code>; otherwise, -substitute an empty string. - - </dl> - - <p>Although these are similar to shell substitutions, no other -substitutions are currently supported, and <code>replacement</code> in the -above cannot contain more substitutions. - - <p>Note: like python, you can either do positional-argument interpolation -<em>or</em> keyword-argument interpolation, not both. Thus you cannot use -a string like <code>WithProperties("foo-%(revision)s-%s", "branch")</code>. - -<h3 class="heading">Common Build Properties</h3> - -<p>The following build properties are set when the build is started, and -are available to all steps. - - <dl> -<dt><code>branch</code><dd> -This comes from the build's SourceStamp, and describes which branch is -being checked out. This will be <code>None</code> (which interpolates into -<code>WithProperties</code> as an empty string) if the build is on the -default branch, which is generally the trunk. Otherwise it will be a -string like “branches/beta1.4”. The exact syntax depends upon the VC -system being used. - - <br><dt><code>revision</code><dd> -This also comes from the SourceStamp, and is the revision of the source code -tree that was requested from the VC system. When a build is requested of a -specific revision (as is generally the case when the build is triggered by -Changes), this will contain the revision specification. This is always a -string, although the syntax depends upon the VC system in use: for SVN it is an -integer, for Mercurial it is a short string, for Darcs it is a rather large -string, etc. - - <p>If the “force build” button was pressed, the revision will be <code>None</code>, -which means to use the most recent revision available. This is a “trunk -build”. This will be interpolated as an empty string. - - <br><dt><code>got_revision</code><dd> -This is set when a Source step checks out the source tree, and -provides the revision that was actually obtained from the VC system. -In general this should be the same as <code>revision</code>, except for -trunk builds, where <code>got_revision</code> indicates what revision was -current when the checkout was performed. This can be used to rebuild -the same source code later. - - <p>Note that for some VC systems (Darcs in particular), the revision is a -large string containing newlines, and is not suitable for interpolation -into a filename. - - <br><dt><code>buildername</code><dd> -This is a string that indicates which Builder the build was a part of. -The combination of buildername and buildnumber uniquely identify a -build. - - <br><dt><code>buildnumber</code><dd> -Each build gets a number, scoped to the Builder (so the first build -performed on any given Builder will have a build number of 0). This -integer property contains the build's number. - - <br><dt><code>slavename</code><dd> -This is a string which identifies which buildslave the build is -running on. - - <br><dt><code>scheduler</code><dd> -If the build was started from a scheduler, then this property will -contain the name of that scheduler. - - </dl> - -<div class="node"> -<p><hr> -<a name="Source-Checkout"></a> -Next: <a rel="next" accesskey="n" href="#ShellCommand">ShellCommand</a>, -Previous: <a rel="previous" accesskey="p" href="#Using-Build-Properties">Using Build Properties</a>, -Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> - -</div> - -<h4 class="subsection">6.1.3 Source Checkout</h4> - -<p>The first step of any build is typically to acquire the source code -from which the build will be performed. There are several classes to -handle this, one for each of the different source control system that -Buildbot knows about. For a description of how Buildbot treats source -control in general, see <a href="#Version-Control-Systems">Version Control Systems</a>. - - <p>All source checkout steps accept some common parameters to control how -they get the sources and where they should be placed. The remaining -per-VC-system parameters are mostly to specify where exactly the -sources are coming from. - - <dl> -<dt><code>mode</code><dd> -a string describing the kind of VC operation that is desired. Defaults -to <code>update</code>. - - <dl> -<dt><code>update</code><dd>specifies that the CVS checkout/update should be performed directly -into the workdir. Each build is performed in the same directory, -allowing for incremental builds. This minimizes disk space, bandwidth, -and CPU time. However, it may encounter problems if the build process -does not handle dependencies properly (sometimes you must do a “clean -build” to make sure everything gets compiled), or if source files are -deleted but generated files can influence test behavior (e.g. python's -.pyc files), or when source directories are deleted but generated -files prevent CVS from removing them. Builds ought to be correct -regardless of whether they are done “from scratch” or incrementally, -but it is useful to test both kinds: this mode exercises the -incremental-build style. - - <br><dt><code>copy</code><dd>specifies that the CVS workspace should be maintained in a separate -directory (called the 'copydir'), using checkout or update as -necessary. For each build, a new workdir is created with a copy of the -source tree (rm -rf workdir; cp -r copydir workdir). This doubles the -disk space required, but keeps the bandwidth low (update instead of a -full checkout). A full 'clean' build is performed each time. This -avoids any generated-file build problems, but is still occasionally -vulnerable to CVS problems such as a repository being manually -rearranged, causing CVS errors on update which are not an issue with a -full checkout. - - <!-- TODO: something is screwy about this, revisit. Is it the source --> - <!-- directory or the working directory that is deleted each time? --> - <br><dt><code>clobber</code><dd>specifes that the working directory should be deleted each time, -necessitating a full checkout for each build. This insures a clean -build off a complete checkout, avoiding any of the problems described -above. This mode exercises the “from-scratch” build style. - - <br><dt><code>export</code><dd>this is like <code>clobber</code>, except that the 'cvs export' command is -used to create the working directory. This command removes all CVS -metadata files (the CVS/ directories) from the tree, which is -sometimes useful for creating source tarballs (to avoid including the -metadata in the tar file). -</dl> - - <br><dt><code>workdir</code><dd>like all Steps, this indicates the directory where the build will take -place. Source Steps are special in that they perform some operations -outside of the workdir (like creating the workdir itself). - - <br><dt><code>alwaysUseLatest</code><dd>if True, bypass the usual “update to the last Change” behavior, and -always update to the latest changes instead. - - <br><dt><code>retry</code><dd>If set, this specifies a tuple of <code>(delay, repeats)</code> which means -that when a full VC checkout fails, it should be retried up to -<var>repeats</var> times, waiting <var>delay</var> seconds between attempts. If -you don't provide this, it defaults to <code>None</code>, which means VC -operations should not be retried. This is provided to make life easier -for buildslaves which are stuck behind poor network connections. - - </dl> - - <p>My habit as a developer is to do a <code>cvs update</code> and <code>make</code> each -morning. Problems can occur, either because of bad code being checked in, or -by incomplete dependencies causing a partial rebuild to fail where a -complete from-scratch build might succeed. A quick Builder which emulates -this incremental-build behavior would use the <code>mode='update'</code> -setting. - - <p>On the other hand, other kinds of dependency problems can cause a clean -build to fail where a partial build might succeed. This frequently results -from a link step that depends upon an object file that was removed from a -later version of the tree: in the partial tree, the object file is still -around (even though the Makefiles no longer know how to create it). - - <p>“official” builds (traceable builds performed from a known set of -source revisions) are always done as clean builds, to make sure it is -not influenced by any uncontrolled factors (like leftover files from a -previous build). A “full” Builder which behaves this way would want -to use the <code>mode='clobber'</code> setting. - - <p>Each VC system has a corresponding source checkout class: their -arguments are described on the following pages. - -<ul class="menu"> -<li><a accesskey="1" href="#CVS">CVS</a> -<li><a accesskey="2" href="#SVN">SVN</a> -<li><a accesskey="3" href="#Darcs">Darcs</a> -<li><a accesskey="4" href="#Mercurial">Mercurial</a> -<li><a accesskey="5" href="#Arch">Arch</a> -<li><a accesskey="6" href="#Bazaar">Bazaar</a> -<li><a accesskey="7" href="#Bzr">Bzr</a> -<li><a accesskey="8" href="#P4">P4</a> -<li><a accesskey="9" href="#Git">Git</a> -</ul> - -<div class="node"> -<p><hr> -<a name="CVS"></a> -Next: <a rel="next" accesskey="n" href="#SVN">SVN</a>, -Previous: <a rel="previous" accesskey="p" href="#Source-Checkout">Source Checkout</a>, -Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> - -</div> - -<h5 class="subsubsection">6.1.3.1 CVS</h5> - -<p><a name="index-CVS-Checkout-53"></a><a name="index-buildbot_002esteps_002esource_002eCVS-54"></a> - - <p>The <code>CVS</code> build step performs a <a href="http://www.nongnu.org/cvs/">CVS</a> checkout or update. It takes the following arguments: - - <dl> -<dt><code>cvsroot</code><dd>(required): specify the CVSROOT value, which points to a CVS -repository, probably on a remote machine. For example, the cvsroot -value you would use to get a copy of the Buildbot source code is -<code>:pserver:anonymous@cvs.sourceforge.net:/cvsroot/buildbot</code> - - <br><dt><code>cvsmodule</code><dd>(required): specify the cvs <code>module</code>, which is generally a -subdirectory of the CVSROOT. The cvsmodule for the Buildbot source -code is <code>buildbot</code>. - - <br><dt><code>branch</code><dd>a string which will be used in a <code>-r</code> argument. This is most -useful for specifying a branch to work on. Defaults to <code>HEAD</code>. - - <br><dt><code>global_options</code><dd>a list of flags to be put before the verb in the CVS command. - - <br><dt><code>checkoutDelay</code><dd>if set, the number of seconds to put between the timestamp of the last -known Change and the value used for the <code>-D</code> option. Defaults to -half of the parent Build's treeStableTimer. - - </dl> - -<div class="node"> -<p><hr> -<a name="SVN"></a> -Next: <a rel="next" accesskey="n" href="#Darcs">Darcs</a>, -Previous: <a rel="previous" accesskey="p" href="#CVS">CVS</a>, -Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> - -</div> - -<h5 class="subsubsection">6.1.3.2 SVN</h5> - -<p><a name="index-SVN-Checkout-55"></a><a name="index-buildbot_002esteps_002esource_002eSVN-56"></a> - - <p>The <code>SVN</code> build step performs a -<a href="http://subversion.tigris.org">Subversion</a> checkout or update. -There are two basic ways of setting up the checkout step, depending -upon whether you are using multiple branches or not. - - <p>If all of your builds use the same branch, then you should create the -<code>SVN</code> step with the <code>svnurl</code> argument: - - <dl> -<dt><code>svnurl</code><dd>(required): this specifies the <code>URL</code> argument that will be given -to the <code>svn checkout</code> command. It dictates both where the -repository is located and which sub-tree should be extracted. In this -respect, it is like a combination of the CVS <code>cvsroot</code> and -<code>cvsmodule</code> arguments. For example, if you are using a remote -Subversion repository which is accessible through HTTP at a URL of -<code>http://svn.example.com/repos</code>, and you wanted to check out the -<code>trunk/calc</code> sub-tree, you would use -<code>svnurl="http://svn.example.com/repos/trunk/calc"</code> as an argument -to your <code>SVN</code> step. -</dl> - - <p>If, on the other hand, you are building from multiple branches, then -you should create the <code>SVN</code> step with the <code>baseURL</code> and -<code>defaultBranch</code> arguments instead: - - <dl> -<dt><code>baseURL</code><dd>(required): this specifies the base repository URL, to which a branch -name will be appended. It should probably end in a slash. - - <br><dt><code>defaultBranch</code><dd>this specifies the name of the branch to use when a Build does not -provide one of its own. This will be appended to <code>baseURL</code> to -create the string that will be passed to the <code>svn checkout</code> -command. - - <br><dt><code>username</code><dd>if specified, this will be passed to the <code>svn</code> binary with a -<code>--username</code> option. - - <br><dt><code>password</code><dd>if specified, this will be passed to the <code>svn</code> binary with a -<code>--password</code> option. The password itself will be suitably obfuscated in -the logs. - - </dl> - - <p>If you are using branches, you must also make sure your -<code>ChangeSource</code> will report the correct branch names. - -<h3 class="heading">branch example</h3> - -<p>Let's suppose that the “MyProject” repository uses branches for the -trunk, for various users' individual development efforts, and for -several new features that will require some amount of work (involving -multiple developers) before they are ready to merge onto the trunk. -Such a repository might be organized as follows: - -<pre class="example"> svn://svn.example.org/MyProject/trunk - svn://svn.example.org/MyProject/branches/User1/foo - svn://svn.example.org/MyProject/branches/User1/bar - svn://svn.example.org/MyProject/branches/User2/baz - svn://svn.example.org/MyProject/features/newthing - svn://svn.example.org/MyProject/features/otherthing -</pre> - <p>Further assume that we want the Buildbot to run tests against the -trunk and against all the feature branches (i.e., do a -checkout/compile/build of branch X when a file has been changed on -branch X, when X is in the set [trunk, features/newthing, -features/otherthing]). We do not want the Buildbot to automatically -build any of the user branches, but it should be willing to build a -user branch when explicitly requested (most likely by the user who -owns that branch). - - <p>There are three things that need to be set up to accomodate this -system. The first is a ChangeSource that is capable of identifying the -branch which owns any given file. This depends upon a user-supplied -function, in an external program that runs in the SVN commit hook and -connects to the buildmaster's <code>PBChangeSource</code> over a TCP -connection. (you can use the “<code>buildbot sendchange</code>” utility -for this purpose, but you will still need an external program to -decide what value should be passed to the <code>--branch=</code> argument). -For example, a change to a file with the SVN url of -“svn://svn.example.org/MyProject/features/newthing/src/foo.c” should -be broken down into a Change instance with -<code>branch='features/newthing'</code> and <code>file='src/foo.c'</code>. - - <p>The second piece is an <code>AnyBranchScheduler</code> which will pay -attention to the desired branches. It will not pay attention to the -user branches, so it will not automatically start builds in response -to changes there. The AnyBranchScheduler class requires you to -explicitly list all the branches you want it to use, but it would not -be difficult to write a subclass which used -<code>branch.startswith('features/'</code> to remove the need for this -explicit list. Or, if you want to build user branches too, you can use -AnyBranchScheduler with <code>branches=None</code> to indicate that you want -it to pay attention to all branches. - - <p>The third piece is an <code>SVN</code> checkout step that is configured to -handle the branches correctly, with a <code>baseURL</code> value that -matches the way the ChangeSource splits each file's URL into base, -branch, and file. - -<pre class="example"> from buildbot.changes.pb import PBChangeSource - from buildbot.scheduler import AnyBranchScheduler - from buildbot.process import source, factory - from buildbot.steps import source, shell - - c['change_source'] = PBChangeSource() - s1 = AnyBranchScheduler('main', - ['trunk', 'features/newthing', 'features/otherthing'], - 10*60, ['test-i386', 'test-ppc']) - c['schedulers'] = [s1] - - f = factory.BuildFactory() - f.addStep(source.SVN(mode='update', - baseURL='svn://svn.example.org/MyProject/', - defaultBranch='trunk')) - f.addStep(shell.Compile(command="make all")) - f.addStep(shell.Test(command="make test")) - - c['builders'] = [ - {'name':'test-i386', 'slavename':'bot-i386', 'builddir':'test-i386', - 'factory':f }, - {'name':'test-ppc', 'slavename':'bot-ppc', 'builddir':'test-ppc', - 'factory':f }, - ] -</pre> - <p>In this example, when a change arrives with a <code>branch</code> attribute -of “trunk”, the resulting build will have an SVN step that -concatenates “svn://svn.example.org/MyProject/” (the baseURL) with -“trunk” (the branch name) to get the correct svn command. If the -“newthing” branch has a change to “src/foo.c”, then the SVN step -will concatenate “svn://svn.example.org/MyProject/” with -“features/newthing” to get the svnurl for checkout. - -<div class="node"> -<p><hr> -<a name="Darcs"></a> -Next: <a rel="next" accesskey="n" href="#Mercurial">Mercurial</a>, -Previous: <a rel="previous" accesskey="p" href="#SVN">SVN</a>, -Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> - -</div> - -<h5 class="subsubsection">6.1.3.3 Darcs</h5> - -<p><a name="index-Darcs-Checkout-57"></a><a name="index-buildbot_002esteps_002esource_002eDarcs-58"></a> - - <p>The <code>Darcs</code> build step performs a -<a href="http://darcs.net/">Darcs</a> checkout or update. - - <p>Like See <a href="#SVN">SVN</a>, this step can either be configured to always check -out a specific tree, or set up to pull from a particular branch that -gets specified separately for each build. Also like SVN, the -repository URL given to Darcs is created by concatenating a -<code>baseURL</code> with the branch name, and if no particular branch is -requested, it uses a <code>defaultBranch</code>. The only difference in -usage is that each potential Darcs repository URL must point to a -fully-fledged repository, whereas SVN URLs usually point to sub-trees -of the main Subversion repository. In other words, doing an SVN -checkout of <code>baseURL</code> is legal, but silly, since you'd probably -wind up with a copy of every single branch in the whole repository. -Doing a Darcs checkout of <code>baseURL</code> is just plain wrong, since -the parent directory of a collection of Darcs repositories is not -itself a valid repository. - - <p>The Darcs step takes the following arguments: - - <dl> -<dt><code>repourl</code><dd>(required unless <code>baseURL</code> is provided): the URL at which the -Darcs source repository is available. - - <br><dt><code>baseURL</code><dd>(required unless <code>repourl</code> is provided): the base repository URL, -to which a branch name will be appended. It should probably end in a -slash. - - <br><dt><code>defaultBranch</code><dd>(allowed if and only if <code>baseURL</code> is provided): this specifies -the name of the branch to use when a Build does not provide one of its -own. This will be appended to <code>baseURL</code> to create the string that -will be passed to the <code>darcs get</code> command. -</dl> - -<div class="node"> -<p><hr> -<a name="Mercurial"></a> -Next: <a rel="next" accesskey="n" href="#Arch">Arch</a>, -Previous: <a rel="previous" accesskey="p" href="#Darcs">Darcs</a>, -Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> - -</div> - -<h5 class="subsubsection">6.1.3.4 Mercurial</h5> - -<p><a name="index-Mercurial-Checkout-59"></a><a name="index-buildbot_002esteps_002esource_002eMercurial-60"></a> - - <p>The <code>Mercurial</code> build step performs a -<a href="http://selenic.com/mercurial">Mercurial</a> (aka “hg”) checkout -or update. - - <p>Branches are handled just like See <a href="#Darcs">Darcs</a>. - - <p>The Mercurial step takes the following arguments: - - <dl> -<dt><code>repourl</code><dd>(required unless <code>baseURL</code> is provided): the URL at which the -Mercurial source repository is available. - - <br><dt><code>baseURL</code><dd>(required unless <code>repourl</code> is provided): the base repository URL, -to which a branch name will be appended. It should probably end in a -slash. - - <br><dt><code>defaultBranch</code><dd>(allowed if and only if <code>baseURL</code> is provided): this specifies -the name of the branch to use when a Build does not provide one of its -own. This will be appended to <code>baseURL</code> to create the string that -will be passed to the <code>hg clone</code> command. -</dl> - -<div class="node"> -<p><hr> -<a name="Arch"></a> -Next: <a rel="next" accesskey="n" href="#Bazaar">Bazaar</a>, -Previous: <a rel="previous" accesskey="p" href="#Mercurial">Mercurial</a>, -Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> - -</div> - -<h5 class="subsubsection">6.1.3.5 Arch</h5> - -<p><a name="index-Arch-Checkout-61"></a><a name="index-buildbot_002esteps_002esource_002eArch-62"></a> - - <p>The <code>Arch</code> build step performs an <a href="http://gnuarch.org/">Arch</a> checkout or update using the <code>tla</code> client. It takes the -following arguments: - - <dl> -<dt><code>url</code><dd>(required): this specifies the URL at which the Arch source archive is -available. - - <br><dt><code>version</code><dd>(required): this specifies which “development line” (like a branch) -should be used. This provides the default branch name, but individual -builds may specify a different one. - - <br><dt><code>archive</code><dd>(optional): Each repository knows its own archive name. If this -parameter is provided, it must match the repository's archive name. -The parameter is accepted for compatibility with the <code>Bazaar</code> -step, below. - - </dl> - -<div class="node"> -<p><hr> -<a name="Bazaar"></a> -Next: <a rel="next" accesskey="n" href="#Bzr">Bzr</a>, -Previous: <a rel="previous" accesskey="p" href="#Arch">Arch</a>, -Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> - -</div> - -<h5 class="subsubsection">6.1.3.6 Bazaar</h5> - -<p><a name="index-Bazaar-Checkout-63"></a><a name="index-buildbot_002esteps_002esource_002eBazaar-64"></a> - - <p><code>Bazaar</code> is an alternate implementation of the Arch VC system, -which uses a client named <code>baz</code>. The checkout semantics are just -different enough from <code>tla</code> that there is a separate BuildStep for -it. - - <p>It takes exactly the same arguments as <code>Arch</code>, except that the -<code>archive=</code> parameter is required. (baz does not emit the archive -name when you do <code>baz register-archive</code>, so we must provide it -ourselves). - -<div class="node"> -<p><hr> -<a name="Bzr"></a> -Next: <a rel="next" accesskey="n" href="#P4">P4</a>, -Previous: <a rel="previous" accesskey="p" href="#Bazaar">Bazaar</a>, -Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> - -</div> - -<h5 class="subsubsection">6.1.3.7 Bzr</h5> - -<p><a name="index-Bzr-Checkout-65"></a><a name="index-buildbot_002esteps_002esource_002eBzr-66"></a> -<code>bzr</code> is a descendant of Arch/Baz, and is frequently referred to -as simply “Bazaar”. The repository-vs-workspace model is similar to -Darcs, but it uses a strictly linear sequence of revisions (one -history per branch) like Arch. Branches are put in subdirectories. -This makes it look very much like Mercurial, so it takes the same -arguments: - - <dl> -<dt><code>repourl</code><dd>(required unless <code>baseURL</code> is provided): the URL at which the -Bzr source repository is available. - - <br><dt><code>baseURL</code><dd>(required unless <code>repourl</code> is provided): the base repository URL, -to which a branch name will be appended. It should probably end in a -slash. - - <br><dt><code>defaultBranch</code><dd>(allowed if and only if <code>baseURL</code> is provided): this specifies -the name of the branch to use when a Build does not provide one of its -own. This will be appended to <code>baseURL</code> to create the string that -will be passed to the <code>bzr checkout</code> command. -</dl> - -<div class="node"> -<p><hr> -<a name="P4"></a> -Next: <a rel="next" accesskey="n" href="#Git">Git</a>, -Previous: <a rel="previous" accesskey="p" href="#Bzr">Bzr</a>, -Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> - -</div> - -<h5 class="subsubsection">6.1.3.8 P4</h5> - -<p><a name="index-Perforce-Update-67"></a><a name="index-buildbot_002esteps_002esource_002eP4-68"></a><!-- TODO @bsindex buildbot.steps.source.P4Sync --> - - <p>The <code>P4</code> build step creates a <a href="http://www.perforce.com/">Perforce</a> client specification and performs an update. - - <dl> -<dt><code>p4base</code><dd>A view into the Perforce depot without branch name or trailing "...". -Typically "//depot/proj/". -<br><dt><code>defaultBranch</code><dd>A branch name to append on build requests if none is specified. -Typically "trunk". -<br><dt><code>p4port</code><dd>(optional): the host:port string describing how to get to the P4 Depot -(repository), used as the -p argument for all p4 commands. -<br><dt><code>p4user</code><dd>(optional): the Perforce user, used as the -u argument to all p4 -commands. -<br><dt><code>p4passwd</code><dd>(optional): the Perforce password, used as the -p argument to all p4 -commands. -<br><dt><code>p4extra_views</code><dd>(optional): a list of (depotpath, clientpath) tuples containing extra -views to be mapped into the client specification. Both will have -"/..." appended automatically. The client name and source directory -will be prepended to the client path. -<br><dt><code>p4client</code><dd>(optional): The name of the client to use. In mode='copy' and -mode='update', it's particularly important that a unique name is used -for each checkout directory to avoid incorrect synchronization. For -this reason, Python percent substitution will be performed on this value -to replace %(slave)s with the slave name and %(builder)s with the -builder name. The default is "buildbot_%(slave)s_%(build)s". -</dl> - -<div class="node"> -<p><hr> -<a name="Git"></a> -Previous: <a rel="previous" accesskey="p" href="#P4">P4</a>, -Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> - -</div> - -<h5 class="subsubsection">6.1.3.9 Git</h5> - -<p><a name="index-Git-Checkout-69"></a><a name="index-buildbot_002esteps_002esource_002eGit-70"></a> -The <code>Git</code> build step clones or updates a <a href="http://git.or.cz/">Git</a> repository and checks out the specified branch or revision. Note -that the buildbot supports Git version 1.2.0 and later: earlier -versions (such as the one shipped in Ubuntu 'Dapper') do not support -the <samp><span class="command">git init</span></samp> command that the buildbot uses. - - <p>The Git step takes the following arguments: - - <dl> -<dt><code>repourl</code><dd>(required): the URL of the upstream Git repository. - - <br><dt><code>branch</code><dd>(optional): this specifies the name of the branch to use when a Build -does not provide one of its own. If this this parameter is not -specified, and the Build does not provide a branch, the “master” -branch will be used. -</dl> - -<div class="node"> -<p><hr> -<a name="ShellCommand"></a> -Next: <a rel="next" accesskey="n" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>, -Previous: <a rel="previous" accesskey="p" href="#Source-Checkout">Source Checkout</a>, -Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> - -</div> - -<h4 class="subsection">6.1.4 ShellCommand</h4> - -<p><a name="index-buildbot_002esteps_002eshell_002eShellCommand-71"></a><!-- TODO @bsindex buildbot.steps.shell.TreeSize --> - - <p>This is a useful base class for just about everything you might want -to do during a build (except for the initial source checkout). It runs -a single command in a child shell on the buildslave. All stdout/stderr -is recorded into a LogFile. The step finishes with a status of FAILURE -if the command's exit code is non-zero, otherwise it has a status of -SUCCESS. - - <p>The preferred way to specify the command is with a list of argv strings, -since this allows for spaces in filenames and avoids doing any fragile -shell-escaping. You can also specify the command with a single string, in -which case the string is given to '/bin/sh -c COMMAND' for parsing. - - <p>On Windows, commands are run via <code>cmd.exe /c</code> which works well. However, -if you're running a batch file, the error level does not get propagated -correctly unless you add 'call' before your batch file's name: -<code>cmd=['call', 'myfile.bat', ...]</code>. - - <p>All ShellCommands are run by default in the “workdir”, which -defaults to the “<samp><span class="file">build</span></samp>” subdirectory of the slave builder's -base directory. The absolute path of the workdir will thus be the -slave's basedir (set as an option to <code>buildbot create-slave</code>, -see <a href="#Creating-a-buildslave">Creating a buildslave</a>) plus the builder's basedir (set in the -builder's <code>c['builddir']</code> key in master.cfg) plus the workdir -itself (a class-level attribute of the BuildFactory, defaults to -“<samp><span class="file">build</span></samp>”). - - <p><code>ShellCommand</code> arguments: - - <dl> -<dt><code>command</code><dd>a list of strings (preferred) or single string (discouraged) which -specifies the command to be run. A list of strings is preferred -because it can be used directly as an argv array. Using a single -string (with embedded spaces) requires the buildslave to pass the -string to /bin/sh for interpretation, which raises all sorts of -difficult questions about how to escape or interpret shell -metacharacters. - - <br><dt><code>env</code><dd>a dictionary of environment strings which will be added to the child -command's environment. For example, to run tests with a different i18n -language setting, you might use - - <pre class="example"> f.addStep(ShellCommand(command=["make", "test"], - env={'LANG': 'fr_FR'})) -</pre> - <p>These variable settings will override any existing ones in the -buildslave's environment or the environment specified in the -Builder. The exception is PYTHONPATH, which is merged -with (actually prepended to) any existing $PYTHONPATH setting. The -value is treated as a list of directories to prepend, and a single -string is treated like a one-item list. For example, to prepend both -<samp><span class="file">/usr/local/lib/python2.3</span></samp> and <samp><span class="file">/home/buildbot/lib/python</span></samp> -to any existing $PYTHONPATH setting, you would do something like the -following: - - <pre class="example"> f.addStep(ShellCommand( - command=["make", "test"], - env={'PYTHONPATH': ["/usr/local/lib/python2.3", - "/home/buildbot/lib/python"] })) -</pre> - <br><dt><code>want_stdout</code><dd>if False, stdout from the child process is discarded rather than being -sent to the buildmaster for inclusion in the step's LogFile. - - <br><dt><code>want_stderr</code><dd>like <code>want_stdout</code> but for stderr. Note that commands run through -a PTY do not have separate stdout/stderr streams: both are merged into -stdout. - - <br><dt><code>usePTY</code><dd>Should this command be run in a <code>pty</code>? The default is to observe the -configuration of the client (see <a href="#Buildslave-Options">Buildslave Options</a>), but specifying -<code>True</code> or <code>False</code> here will override the default. - - <p>The advantage of using a PTY is that “grandchild” processes are more likely -to be cleaned up if the build is interrupted or times out (since it enables the -use of a “process group” in which all child processes will be placed). The -disadvantages: some forms of Unix have problems with PTYs, some of your unit -tests may behave differently when run under a PTY (generally those which check -to see if they are being run interactively), and PTYs will merge the stdout and -stderr streams into a single output stream (which means the red-vs-black -coloring in the logfiles will be lost). - - <br><dt><code>logfiles</code><dd>Sometimes commands will log interesting data to a local file, rather -than emitting everything to stdout or stderr. For example, Twisted's -“trial” command (which runs unit tests) only presents summary -information to stdout, and puts the rest into a file named -<samp><span class="file">_trial_temp/test.log</span></samp>. It is often useful to watch these files -as the command runs, rather than using <samp><span class="command">/bin/cat</span></samp> to dump -their contents afterwards. - - <p>The <code>logfiles=</code> argument allows you to collect data from these -secondary logfiles in near-real-time, as the step is running. It -accepts a dictionary which maps from a local Log name (which is how -the log data is presented in the build results) to a remote filename -(interpreted relative to the build's working directory). Each named -file will be polled on a regular basis (every couple of seconds) as -the build runs, and any new text will be sent over to the buildmaster. - - <pre class="example"> f.addStep(ShellCommand( - command=["make", "test"], - logfiles={"triallog": "_trial_temp/test.log"})) -</pre> - <br><dt><code>timeout</code><dd>if the command fails to produce any output for this many seconds, it -is assumed to be locked up and will be killed. - - <br><dt><code>description</code><dd>This will be used to describe the command (on the Waterfall display) -while the command is still running. It should be a single -imperfect-tense verb, like “compiling” or “testing”. The preferred -form is a list of short strings, which allows the HTML Waterfall -display to create narrower columns by emitting a <br> tag between each -word. You may also provide a single string. - - <br><dt><code>descriptionDone</code><dd>This will be used to describe the command once it has finished. A -simple noun like “compile” or “tests” should be used. Like -<code>description</code>, this may either be a list of short strings or a -single string. - - <p>If neither <code>description</code> nor <code>descriptionDone</code> are set, the -actual command arguments will be used to construct the description. -This may be a bit too wide to fit comfortably on the Waterfall -display. - - <pre class="example"> f.addStep(ShellCommand(command=["make", "test"], - description=["testing"], - descriptionDone=["tests"])) -</pre> - <br><dt><code>logEnviron</code><dd>If this option is true (the default), then the step's logfile will describe the -environment variables on the slave. In situations where the environment is not -relevant and is long, it may be easier to set <code>logEnviron=False</code>. - - </dl> - -<div class="node"> -<p><hr> -<a name="Simple-ShellCommand-Subclasses"></a> -Next: <a rel="next" accesskey="n" href="#Python-BuildSteps">Python BuildSteps</a>, -Previous: <a rel="previous" accesskey="p" href="#ShellCommand">ShellCommand</a>, -Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> - -</div> - -<h4 class="subsection">6.1.5 Simple ShellCommand Subclasses</h4> - -<p>Several subclasses of ShellCommand are provided as starting points for -common build steps. These are all very simple: they just override a few -parameters so you don't have to specify them yourself, making the master.cfg -file less verbose. - -<ul class="menu"> -<li><a accesskey="1" href="#Configure">Configure</a> -<li><a accesskey="2" href="#Compile">Compile</a> -<li><a accesskey="3" href="#Test">Test</a> -<li><a accesskey="4" href="#TreeSize">TreeSize</a> -<li><a accesskey="5" href="#PerlModuleTest">PerlModuleTest</a> -<li><a accesskey="6" href="#SetProperty">SetProperty</a> -</ul> - -<div class="node"> -<p><hr> -<a name="Configure"></a> -Next: <a rel="next" accesskey="n" href="#Compile">Compile</a>, -Previous: <a rel="previous" accesskey="p" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>, -Up: <a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a> - -</div> - -<h5 class="subsubsection">6.1.5.1 Configure</h5> - -<p><a name="index-buildbot_002esteps_002eshell_002eConfigure-72"></a> -This is intended to handle the <code>./configure</code> step from -autoconf-style projects, or the <code>perl Makefile.PL</code> step from perl -MakeMaker.pm-style modules. The default command is <code>./configure</code> -but you can change this by providing a <code>command=</code> parameter. - -<div class="node"> -<p><hr> -<a name="Compile"></a> -Next: <a rel="next" accesskey="n" href="#Test">Test</a>, -Previous: <a rel="previous" accesskey="p" href="#Configure">Configure</a>, -Up: <a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a> - -</div> - -<h5 class="subsubsection">6.1.5.2 Compile</h5> - -<p><a name="index-buildbot_002esteps_002eshell_002eCompile-73"></a> -This is meant to handle compiling or building a project written in C. -The default command is <code>make all</code>. When the compile is finished, -the log file is scanned for GCC warning messages, a summary log is -created with any problems that were seen, and the step is marked as -WARNINGS if any were discovered. The number of warnings is stored in a -Build Property named “warnings-count”, which is accumulated over all -Compile steps (so if two warnings are found in one step, and three are -found in another step, the overall build will have a -“warnings-count” property of 5. - - <p>The default regular expression used to detect a warning is -<code>'.*warning[: ].*'</code> , which is fairly liberal and may cause -false-positives. To use a different regexp, provide a -<code>warningPattern=</code> argument, or use a subclass which sets the -<code>warningPattern</code> attribute: - -<pre class="example"> f.addStep(Compile(command=["make", "test"], - warningPattern="^Warning: ")) -</pre> - <p>The <code>warningPattern=</code> can also be a pre-compiled python regexp -object: this makes it possible to add flags like <code>re.I</code> (to use -case-insensitive matching). - - <p>(TODO: this step needs to be extended to look for GCC error messages -as well, and collect them into a separate logfile, along with the -source code filenames involved). - -<div class="node"> -<p><hr> -<a name="Test"></a> -Next: <a rel="next" accesskey="n" href="#TreeSize">TreeSize</a>, -Previous: <a rel="previous" accesskey="p" href="#Compile">Compile</a>, -Up: <a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a> - -</div> - -<h5 class="subsubsection">6.1.5.3 Test</h5> - -<p><a name="index-buildbot_002esteps_002eshell_002eTest-74"></a> -This is meant to handle unit tests. The default command is <code>make -test</code>, and the <code>warnOnFailure</code> flag is set. - -<div class="node"> -<p><hr> -<a name="TreeSize"></a> -Next: <a rel="next" accesskey="n" href="#PerlModuleTest">PerlModuleTest</a>, -Previous: <a rel="previous" accesskey="p" href="#Test">Test</a>, -Up: <a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a> - -</div> - -<h5 class="subsubsection">6.1.5.4 TreeSize</h5> - -<p><a name="index-buildbot_002esteps_002eshell_002eTreeSize-75"></a> -This is a simple command that uses the 'du' tool to measure the size -of the code tree. It puts the size (as a count of 1024-byte blocks, -aka 'KiB' or 'kibibytes') on the step's status text, and sets a build -property named 'tree-size-KiB' with the same value. - -<div class="node"> -<p><hr> -<a name="PerlModuleTest"></a> -Next: <a rel="next" accesskey="n" href="#SetProperty">SetProperty</a>, -Previous: <a rel="previous" accesskey="p" href="#TreeSize">TreeSize</a>, -Up: <a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a> - -</div> - -<h5 class="subsubsection">6.1.5.5 PerlModuleTest</h5> - -<p><a name="index-buildbot_002esteps_002eshell_002ePerlModuleTest-76"></a> -This is a simple command that knows how to run tests of perl modules. -It parses the output to determine the number of tests passed and -failed and total number executed, saving the results for later query. - -<div class="node"> -<p><hr> -<a name="SetProperty"></a> -Previous: <a rel="previous" accesskey="p" href="#PerlModuleTest">PerlModuleTest</a>, -Up: <a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a> - -</div> - -<h5 class="subsubsection">6.1.5.6 SetProperty</h5> - -<p><a name="index-buildbot_002esteps_002eshell_002eSetProperty-77"></a> -This buildstep is similar to ShellCommand, except that it captures the -output of the command into a property. It is usually used like this: - -<pre class="example"> f.addStep(SetProperty(command="uname -a", property="uname")) -</pre> - <p>This runs <code>uname -a</code> and captures its stdout, stripped of leading -and trailing whitespace, in the property "uname". To avoid stripping, -add <code>strip=False</code>. The <code>property</code> argument can be specified -as a <code>WithProperties</code> object. - - <p>The more advanced usage allows you to specify a function to extract -properties from the command output. Here you can use regular -expressions, string interpolation, or whatever you would like. -The function is called with three arguments: the exit status of the -command, its standard output as a string, and its standard error as -a string. It should return a dictionary containing all new properties. - -<pre class="example"> def glob2list(rc, stdout, stderr): - jpgs = [ l.strip() for l in stdout.split('\n') ] - return { 'jpgs' : jpgs } - f.addStep(SetProperty(command="ls -1 *.jpg", extract_fn=glob2list)) -</pre> - <p>Note that any ordering relationship of the contents of stdout and -stderr is lost. For example, given - -<pre class="example"> f.addStep(SetProperty( - command="echo output1; echo error >&2; echo output2", - extract_fn=my_extract)) -</pre> - <p>Then <code>my_extract</code> will see <code>stdout="output1\noutput2\n"</code> -and <code>stderr="error\n"</code>. - -<div class="node"> -<p><hr> -<a name="Python-BuildSteps"></a> -Next: <a rel="next" accesskey="n" href="#Transferring-Files">Transferring Files</a>, -Previous: <a rel="previous" accesskey="p" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>, -Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> - -</div> - -<h4 class="subsection">6.1.6 Python BuildSteps</h4> - -<p>Here are some BuildSteps that are specifcally useful for projects -implemented in Python. - -<ul class="menu"> -<li><a accesskey="1" href="#BuildEPYDoc">BuildEPYDoc</a> -<li><a accesskey="2" href="#PyFlakes">PyFlakes</a> -<li><a accesskey="3" href="#PyLint">PyLint</a> -</ul> - -<div class="node"> -<p><hr> -<a name="BuildEPYDoc"></a> -Next: <a rel="next" accesskey="n" href="#PyFlakes">PyFlakes</a>, -Up: <a rel="up" accesskey="u" href="#Python-BuildSteps">Python BuildSteps</a> - -</div> - -<h5 class="subsubsection">6.1.6.1 BuildEPYDoc</h5> - -<p><a name="index-buildbot_002esteps_002epython_002eBuildEPYDoc-78"></a> -<a href="http://epydoc.sourceforge.net/">epydoc</a> is a tool for generating -API documentation for Python modules from their docstrings. It reads -all the .py files from your source tree, processes the docstrings -therein, and creates a large tree of .html files (or a single .pdf -file). - - <p>The <code>buildbot.steps.python.BuildEPYDoc</code> step will run -<samp><span class="command">epydoc</span></samp> to produce this API documentation, and will count the -errors and warnings from its output. - - <p>You must supply the command line to be used. The default is -<samp><span class="command">make epydocs</span></samp>, which assumes that your project has a Makefile -with an “epydocs” target. You might wish to use something like -<samp><span class="command">epydoc -o apiref source/PKGNAME</span></samp> instead. You might also want -to add <samp><span class="command">--pdf</span></samp> to generate a PDF file instead of a large tree -of HTML files. - - <p>The API docs are generated in-place in the build tree (under the -workdir, in the subdirectory controlled by the “-o” argument). To -make them useful, you will probably have to copy them to somewhere -they can be read. A command like <samp><span class="command">rsync -ad apiref/ -dev.example.com:~public_html/current-apiref/</span></samp> might be useful. You -might instead want to bundle them into a tarball and publish it in the -same place where the generated install tarball is placed. - -<pre class="example"> from buildbot.steps.python import BuildEPYDoc - - ... - f.addStep(BuildEPYDoc(command=["epydoc", "-o", "apiref", "source/mypkg"])) -</pre> - <div class="node"> -<p><hr> -<a name="PyFlakes"></a> -Next: <a rel="next" accesskey="n" href="#PyLint">PyLint</a>, -Previous: <a rel="previous" accesskey="p" href="#BuildEPYDoc">BuildEPYDoc</a>, -Up: <a rel="up" accesskey="u" href="#Python-BuildSteps">Python BuildSteps</a> - -</div> - -<h5 class="subsubsection">6.1.6.2 PyFlakes</h5> - -<p><a name="index-buildbot_002esteps_002epython_002ePyFlakes-79"></a> -<a href="http://divmod.org/trac/wiki/DivmodPyflakes">PyFlakes</a> is a tool -to perform basic static analysis of Python code to look for simple -errors, like missing imports and references of undefined names. It is -like a fast and simple form of the C “lint” program. Other tools -(like pychecker) provide more detailed results but take longer to run. - - <p>The <code>buildbot.steps.python.PyFlakes</code> step will run pyflakes and -count the various kinds of errors and warnings it detects. - - <p>You must supply the command line to be used. The default is -<samp><span class="command">make pyflakes</span></samp>, which assumes you have a top-level Makefile -with a “pyflakes” target. You might want to use something like -<samp><span class="command">pyflakes .</span></samp> or <samp><span class="command">pyflakes src</span></samp>. - -<pre class="example"> from buildbot.steps.python import PyFlakes - - ... - f.addStep(PyFlakes(command=["pyflakes", "src"])) -</pre> - <div class="node"> -<p><hr> -<a name="PyLint"></a> -Previous: <a rel="previous" accesskey="p" href="#PyFlakes">PyFlakes</a>, -Up: <a rel="up" accesskey="u" href="#Python-BuildSteps">Python BuildSteps</a> - -</div> - -<h5 class="subsubsection">6.1.6.3 PyLint</h5> - -<p><a name="index-buildbot_002esteps_002epython_002ePyLint-80"></a> -Similarly, the <code>buildbot.steps.python.PyLint</code> step will run pylint and -analyze the results. - - <p>You must supply the command line to be used. There is no default. - -<pre class="example"> from buildbot.steps.python import PyLint - - ... - f.addStep(PyLint(command=["pylint", "src"])) -</pre> - <div class="node"> -<p><hr> -<a name="Transferring-Files"></a> -Next: <a rel="next" accesskey="n" href="#Steps-That-Run-on-the-Master">Steps That Run on the Master</a>, -Previous: <a rel="previous" accesskey="p" href="#Python-BuildSteps">Python BuildSteps</a>, -Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> - -</div> - -<h4 class="subsection">6.1.7 Transferring Files</h4> - -<p><a name="index-File-Transfer-81"></a><a name="index-buildbot_002esteps_002etransfer_002eFileUpload-82"></a><a name="index-buildbot_002esteps_002etransfer_002eFileDownload-83"></a><a name="index-buildbot_002esteps_002etransfer_002eDirectoryUpload-84"></a> -Most of the work involved in a build will take place on the -buildslave. But occasionally it is useful to do some work on the -buildmaster side. The most basic way to involve the buildmaster is -simply to move a file from the slave to the master, or vice versa. -There are a pair of BuildSteps named <code>FileUpload</code> and -<code>FileDownload</code> to provide this functionality. <code>FileUpload</code> -moves a file <em>up to</em> the master, while <code>FileDownload</code> moves -a file <em>down from</em> the master. - - <p>As an example, let's assume that there is a step which produces an -HTML file within the source tree that contains some sort of generated -project documentation. We want to move this file to the buildmaster, -into a <samp><span class="file">~/public_html</span></samp> directory, so it can be visible to -developers. This file will wind up in the slave-side working directory -under the name <samp><span class="file">docs/reference.html</span></samp>. We want to put it into the -master-side <samp><span class="file">~/public_html/ref.html</span></samp>. - -<pre class="example"> from buildbot.steps.shell import ShellCommand - from buildbot.steps.transfer import FileUpload - - f.addStep(ShellCommand(command=["make", "docs"])) - f.addStep(FileUpload(slavesrc="docs/reference.html", - masterdest="~/public_html/ref.html")) -</pre> - <p>The <code>masterdest=</code> argument will be passed to os.path.expanduser, -so things like “~” will be expanded properly. Non-absolute paths -will be interpreted relative to the buildmaster's base directory. -Likewise, the <code>slavesrc=</code> argument will be expanded and -interpreted relative to the builder's working directory. - - <p>To move a file from the master to the slave, use the -<code>FileDownload</code> command. For example, let's assume that some step -requires a configuration file that, for whatever reason, could not be -recorded in the source code repository or generated on the buildslave -side: - -<pre class="example"> from buildbot.steps.shell import ShellCommand - from buildbot.steps.transfer import FileUpload - - f.addStep(FileDownload(mastersrc="~/todays_build_config.txt", - slavedest="build_config.txt")) - f.addStep(ShellCommand(command=["make", "config"])) -</pre> - <p>Like <code>FileUpload</code>, the <code>mastersrc=</code> argument is interpreted -relative to the buildmaster's base directory, and the -<code>slavedest=</code> argument is relative to the builder's working -directory. If the buildslave is running in <samp><span class="file">~buildslave</span></samp>, and the -builder's “builddir” is something like <samp><span class="file">tests-i386</span></samp>, then the -workdir is going to be <samp><span class="file">~buildslave/tests-i386/build</span></samp>, and a -<code>slavedest=</code> of <samp><span class="file">foo/bar.html</span></samp> will get put in -<samp><span class="file">~buildslave/tests-i386/build/foo/bar.html</span></samp>. Both of these commands -will create any missing intervening directories. - -<h4 class="subheading">Other Parameters</h4> - -<p>The <code>maxsize=</code> argument lets you set a maximum size for the file -to be transferred. This may help to avoid surprises: transferring a -100MB coredump when you were expecting to move a 10kB status file -might take an awfully long time. The <code>blocksize=</code> argument -controls how the file is sent over the network: larger blocksizes are -slightly more efficient but also consume more memory on each end, and -there is a hard-coded limit of about 640kB. - - <p>The <code>mode=</code> argument allows you to control the access permissions -of the target file, traditionally expressed as an octal integer. The -most common value is probably 0755, which sets the “x” executable -bit on the file (useful for shell scripts and the like). The default -value for <code>mode=</code> is None, which means the permission bits will -default to whatever the umask of the writing process is. The default -umask tends to be fairly restrictive, but at least on the buildslave -you can make it less restrictive with a –umask command-line option at -creation time (see <a href="#Buildslave-Options">Buildslave Options</a>). - -<h4 class="subheading">Transfering Directories</h4> - -<p>To transfer complete directories from the buildslave to the master, there -is a BuildStep named <code>DirectoryUpload</code>. It works like <code>FileUpload</code>, -just for directories. However it does not support the <code>maxsize</code>, -<code>blocksize</code> and <code>mode</code> arguments. As an example, let's assume an -generated project documentation, which consists of many files (like the output -of doxygen or epydoc). We want to move the entire documentation to the -buildmaster, into a <code>~/public_html/docs</code> directory. On the slave-side -the directory can be found under <code>docs</code>: - -<pre class="example"> from buildbot.steps.shell import ShellCommand - from buildbot.steps.transfer import DirectoryUpload - - f.addStep(ShellCommand(command=["make", "docs"])) - f.addStep(DirectoryUpload(slavesrc="docs", - masterdest="~/public_html/docs")) -</pre> - <p>The DirectoryUpload step will create all necessary directories and -transfers empty directories, too. - -<div class="node"> -<p><hr> -<a name="Steps-That-Run-on-the-Master"></a> -Next: <a rel="next" accesskey="n" href="#Triggering-Schedulers">Triggering Schedulers</a>, -Previous: <a rel="previous" accesskey="p" href="#Transferring-Files">Transferring Files</a>, -Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> - -</div> - -<h4 class="subsection">6.1.8 Steps That Run on the Master</h4> - -<p>Occasionally, it is useful to execute some task on the master, for example to -create a directory, deploy a build result, or trigger some other centralized -processing. This is possible, in a limited fashion, with the -<code>MasterShellCommand</code> step. - - <p>This step operates similarly to a regular <code>ShellCommand</code>, but executes on -the master, instead of the slave. To be clear, the enclosing <code>Build</code> -object must still have a slave object, just as for any other step – only, in -this step, the slave does not do anything. - - <p>In this example, the step renames a tarball based on the day of the week. - -<pre class="example"> from buildbot.steps.transfer import FileUpload - from buildbot.steps.master import MasterShellCommand - - f.addStep(FileUpload(slavesrc="widgetsoft.tar.gz", - masterdest="/var/buildoutputs/widgetsoft-new.tar.gz")) - f.addStep(MasterShellCommand(command=""" - cd /var/buildoutputs; - mv widgetsoft-new.tar.gz widgetsoft-`date +%a`.tar.gz""")) -</pre> - <div class="node"> -<p><hr> -<a name="Triggering-Schedulers"></a> -Next: <a rel="next" accesskey="n" href="#Writing-New-BuildSteps">Writing New BuildSteps</a>, -Previous: <a rel="previous" accesskey="p" href="#Steps-That-Run-on-the-Master">Steps That Run on the Master</a>, -Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> - -</div> - -<h4 class="subsection">6.1.9 Triggering Schedulers</h4> - -<p>The counterpart to the Triggerable described in section -see <a href="#Triggerable-Scheduler">Triggerable Scheduler</a> is the Trigger BuildStep. - -<pre class="example"> from buildbot.steps.trigger import Trigger - f.addStep(Trigger(schedulerNames=['build-prep'], - waitForFinish=True, - updateSourceStamp=True)) -</pre> - <p>The <code>schedulerNames=</code> argument lists the Triggerables -that should be triggered when this step is executed. Note that -it is possible, but not advisable, to create a cycle where a build -continually triggers itself, because the schedulers are specified -by name. - - <p>If <code>waitForFinish</code> is True, then the step will not finish until -all of the builds from the triggered schedulers have finished. If this -argument is False (the default) or not given, then the buildstep -succeeds immediately after triggering the schedulers. - - <p>If <code>updateSourceStamp</code> is True (the default), then step updates -the SourceStamp given to the Triggerables to include -<code>got_revision</code> (the revision actually used in this build) as -<code>revision</code> (the revision to use in the triggered builds). This is -useful to ensure that all of the builds use exactly the same -SourceStamp, even if other Changes have occurred while the build was -running. - -<div class="node"> -<p><hr> -<a name="Writing-New-BuildSteps"></a> -Previous: <a rel="previous" accesskey="p" href="#Triggering-Schedulers">Triggering Schedulers</a>, -Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> - -</div> - -<h4 class="subsection">6.1.10 Writing New BuildSteps</h4> - -<p>While it is a good idea to keep your build process self-contained in -the source code tree, sometimes it is convenient to put more -intelligence into your Buildbot configuration. One way to do this is -to write a custom BuildStep. Once written, this Step can be used in -the <samp><span class="file">master.cfg</span></samp> file. - - <p>The best reason for writing a custom BuildStep is to better parse the -results of the command being run. For example, a BuildStep that knows -about JUnit could look at the logfiles to determine which tests had -been run, how many passed and how many failed, and then report more -detailed information than a simple <code>rc==0</code> -based “good/bad” -decision. - -<ul class="menu"> -<li><a accesskey="1" href="#Writing-BuildStep-Constructors">Writing BuildStep Constructors</a> -<li><a accesskey="2" href="#BuildStep-LogFiles">BuildStep LogFiles</a> -<li><a accesskey="3" href="#Reading-Logfiles">Reading Logfiles</a> -<li><a accesskey="4" href="#Adding-LogObservers">Adding LogObservers</a> -<li><a accesskey="5" href="#BuildStep-URLs">BuildStep URLs</a> -</ul> - -<div class="node"> -<p><hr> -<a name="Writing-BuildStep-Constructors"></a> -Next: <a rel="next" accesskey="n" href="#BuildStep-LogFiles">BuildStep LogFiles</a>, -Up: <a rel="up" accesskey="u" href="#Writing-New-BuildSteps">Writing New BuildSteps</a> - -</div> - -<h5 class="subsubsection">6.1.10.1 Writing BuildStep Constructors</h5> - -<p>BuildStep classes have some extra equipment, because they are their own -factories. Consider the use of a BuildStep in <samp><span class="file">master.cfg</span></samp>: - -<pre class="example"> f.addStep(MyStep(someopt="stuff", anotheropt=1)) -</pre> - <p>This creates a single instance of class <code>MyStep</code>. However, Buildbot needs -a new object each time the step is executed. this is accomplished by storing -the information required to instantiate a new object in the <code>factory</code> -attribute. When the time comes to construct a new Build, BuildFactory consults -this attribute (via <code>getStepFactory</code>) and instantiates a new step object. - - <p>When writing a new step class, then, keep in mind are that you cannot do -anything "interesting" in the constructor – limit yourself to checking and -storing arguments. To ensure that these arguments are provided to any new -objects, call <code>self.addFactoryArguments</code> with any keyword arguments your -constructor needs. - - <p>Keep a <code>**kwargs</code> argument on the end of your options, and pass that up to -the parent class's constructor. - - <p>The whole thing looks like this: - -<pre class="example"> class Frobinfy(LoggingBuildStep): - def __init__(self, - frob_what="frobee", - frob_how_many=None, - frob_how=None, - **kwargs) - - # check - if frob_how_many is None: - raise TypeError("Frobinfy argument how_many is required") - - # call parent - LoggingBuildStep.__init__(self, **kwargs) - - # and record arguments for later - self.addFactoryArguments( - frob_what=frob_what, - frob_how_many=frob_how_many, - frob_how=frob_how) - - class FastFrobnify(Frobnify): - def __init__(self, - speed=5, - **kwargs) - Frobnify.__init__(self, **kwargs) - self.addFactoryArguments( - speed=speed) -</pre> - <div class="node"> -<p><hr> -<a name="BuildStep-LogFiles"></a> -Next: <a rel="next" accesskey="n" href="#Reading-Logfiles">Reading Logfiles</a>, -Previous: <a rel="previous" accesskey="p" href="#Writing-BuildStep-Constructors">Writing BuildStep Constructors</a>, -Up: <a rel="up" accesskey="u" href="#Writing-New-BuildSteps">Writing New BuildSteps</a> - -</div> - -<h5 class="subsubsection">6.1.10.2 BuildStep LogFiles</h5> - -<p>Each BuildStep has a collection of “logfiles”. Each one has a short -name, like “stdio” or “warnings”. Each LogFile contains an -arbitrary amount of text, usually the contents of some output file -generated during a build or test step, or a record of everything that -was printed to stdout/stderr during the execution of some command. - - <p>These LogFiles are stored to disk, so they can be retrieved later. - - <p>Each can contain multiple “channels”, generally limited to three -basic ones: stdout, stderr, and “headers”. For example, when a -ShellCommand runs, it writes a few lines to the “headers” channel to -indicate the exact argv strings being run, which directory the command -is being executed in, and the contents of the current environment -variables. Then, as the command runs, it adds a lot of “stdout” and -“stderr” messages. When the command finishes, a final “header” -line is added with the exit code of the process. - - <p>Status display plugins can format these different channels in -different ways. For example, the web page shows LogFiles as text/html, -with header lines in blue text, stdout in black, and stderr in red. A -different URL is available which provides a text/plain format, in -which stdout and stderr are collapsed together, and header lines are -stripped completely. This latter option makes it easy to save the -results to a file and run <samp><span class="command">grep</span></samp> or whatever against the -output. - - <p>Each BuildStep contains a mapping (implemented in a python dictionary) -from LogFile name to the actual LogFile objects. Status plugins can -get a list of LogFiles to display, for example, a list of HREF links -that, when clicked, provide the full contents of the LogFile. - -<h3 class="heading">Using LogFiles in custom BuildSteps</h3> - -<p>The most common way for a custom BuildStep to use a LogFile is to -summarize the results of a ShellCommand (after the command has -finished running). For example, a compile step with thousands of lines -of output might want to create a summary of just the warning messages. -If you were doing this from a shell, you would use something like: - -<pre class="example"> grep "warning:" output.log >warnings.log -</pre> - <p>In a custom BuildStep, you could instead create a “warnings” LogFile -that contained the same text. To do this, you would add code to your -<code>createSummary</code> method that pulls lines from the main output log -and creates a new LogFile with the results: - -<pre class="example"> def createSummary(self, log): - warnings = [] - for line in log.readlines(): - if "warning:" in line: - warnings.append() - self.addCompleteLog('warnings', "".join(warnings)) -</pre> - <p>This example uses the <code>addCompleteLog</code> method, which creates a -new LogFile, puts some text in it, and then “closes” it, meaning -that no further contents will be added. This LogFile will appear in -the HTML display under an HREF with the name “warnings”, since that -is the name of the LogFile. - - <p>You can also use <code>addHTMLLog</code> to create a complete (closed) -LogFile that contains HTML instead of plain text. The normal LogFile -will be HTML-escaped if presented through a web page, but the HTML -LogFile will not. At the moment this is only used to present a pretty -HTML representation of an otherwise ugly exception traceback when -something goes badly wrong during the BuildStep. - - <p>In contrast, you might want to create a new LogFile at the beginning -of the step, and add text to it as the command runs. You can create -the LogFile and attach it to the build by calling <code>addLog</code>, which -returns the LogFile object. You then add text to this LogFile by -calling methods like <code>addStdout</code> and <code>addHeader</code>. When you -are done, you must call the <code>finish</code> method so the LogFile can be -closed. It may be useful to create and populate a LogFile like this -from a LogObserver method See <a href="#Adding-LogObservers">Adding LogObservers</a>. - - <p>The <code>logfiles=</code> argument to <code>ShellCommand</code> (see -see <a href="#ShellCommand">ShellCommand</a>) creates new LogFiles and fills them in realtime -by asking the buildslave to watch a actual file on disk. The -buildslave will look for additions in the target file and report them -back to the BuildStep. These additions will be added to the LogFile by -calling <code>addStdout</code>. These secondary LogFiles can be used as the -source of a LogObserver just like the normal “stdio” LogFile. - -<div class="node"> -<p><hr> -<a name="Reading-Logfiles"></a> -Next: <a rel="next" accesskey="n" href="#Adding-LogObservers">Adding LogObservers</a>, -Previous: <a rel="previous" accesskey="p" href="#BuildStep-LogFiles">BuildStep LogFiles</a>, -Up: <a rel="up" accesskey="u" href="#Writing-New-BuildSteps">Writing New BuildSteps</a> - -</div> - -<h5 class="subsubsection">6.1.10.3 Reading Logfiles</h5> - -<p>Once a LogFile has been added to a BuildStep with <code>addLog()</code>, -<code>addCompleteLog()</code>, <code>addHTMLLog()</code>, or <code>logfiles=</code>, -your BuildStep can retrieve it by using <code>getLog()</code>: - -<pre class="example"> class MyBuildStep(ShellCommand): - logfiles = { "nodelog": "_test/node.log" } - - def evaluateCommand(self, cmd): - nodelog = self.getLog("nodelog") - if "STARTED" in nodelog.getText(): - return SUCCESS - else: - return FAILURE -</pre> - <p>For a complete list of the methods you can call on a LogFile, please -see the docstrings on the <code>IStatusLog</code> class in -<samp><span class="file">buildbot/interfaces.py</span></samp>. - -<div class="node"> -<p><hr> -<a name="Adding-LogObservers"></a> -Next: <a rel="next" accesskey="n" href="#BuildStep-URLs">BuildStep URLs</a>, -Previous: <a rel="previous" accesskey="p" href="#Reading-Logfiles">Reading Logfiles</a>, -Up: <a rel="up" accesskey="u" href="#Writing-New-BuildSteps">Writing New BuildSteps</a> - -</div> - -<h5 class="subsubsection">6.1.10.4 Adding LogObservers</h5> - -<p><a name="index-LogObserver-85"></a><a name="index-LogLineObserver-86"></a> -Most shell commands emit messages to stdout or stderr as they operate, -especially if you ask them nicely with a <code>--verbose</code> flag of some -sort. They may also write text to a log file while they run. Your -BuildStep can watch this output as it arrives, to keep track of how -much progress the command has made. You can get a better measure of -progress by counting the number of source files compiled or test cases -run than by merely tracking the number of bytes that have been written -to stdout. This improves the accuracy and the smoothness of the ETA -display. - - <p>To accomplish this, you will need to attach a <code>LogObserver</code> to -one of the log channels, most commonly to the “stdio” channel but -perhaps to another one which tracks a log file. This observer is given -all text as it is emitted from the command, and has the opportunity to -parse that output incrementally. Once the observer has decided that -some event has occurred (like a source file being compiled), it can -use the <code>setProgress</code> method to tell the BuildStep about the -progress that this event represents. - - <p>There are a number of pre-built <code>LogObserver</code> classes that you -can choose from (defined in <code>buildbot.process.buildstep</code>, and of -course you can subclass them to add further customization. The -<code>LogLineObserver</code> class handles the grunt work of buffering and -scanning for end-of-line delimiters, allowing your parser to operate -on complete stdout/stderr lines. (Lines longer than a set maximum -length are dropped; the maximum defaults to 16384 bytes, but you can -change it by calling <code>setMaxLineLength()</code> on your -<code>LogLineObserver</code> instance. Use <code>sys.maxint</code> for effective -infinity.) - - <p>For example, let's take a look at the <code>TrialTestCaseCounter</code>, -which is used by the Trial step to count test cases as they are run. -As Trial executes, it emits lines like the following: - -<pre class="example"> buildbot.test.test_config.ConfigTest.testDebugPassword ... [OK] - buildbot.test.test_config.ConfigTest.testEmpty ... [OK] - buildbot.test.test_config.ConfigTest.testIRC ... [FAIL] - buildbot.test.test_config.ConfigTest.testLocks ... [OK] -</pre> - <p>When the tests are finished, trial emits a long line of “======” and -then some lines which summarize the tests that failed. We want to -avoid parsing these trailing lines, because their format is less -well-defined than the “[OK]” lines. - - <p>The parser class looks like this: - -<pre class="example"> from buildbot.process.buildstep import LogLineObserver - - class TrialTestCaseCounter(LogLineObserver): - _line_re = re.compile(r'^([\w\.]+) \.\.\. \[([^\]]+)\]$') - numTests = 0 - finished = False - - def outLineReceived(self, line): - if self.finished: - return - if line.startswith("=" * 40): - self.finished = True - return - - m = self._line_re.search(line.strip()) - if m: - testname, result = m.groups() - self.numTests += 1 - self.step.setProgress('tests', self.numTests) -</pre> - <p>This parser only pays attention to stdout, since that's where trial -writes the progress lines. It has a mode flag named <code>finished</code> to -ignore everything after the “====” marker, and a scary-looking -regular expression to match each line while hopefully ignoring other -messages that might get displayed as the test runs. - - <p>Each time it identifies a test has been completed, it increments its -counter and delivers the new progress value to the step with -<code>self.step.setProgress</code>. This class is specifically measuring -progress along the “tests” metric, in units of test cases (as -opposed to other kinds of progress like the “output” metric, which -measures in units of bytes). The Progress-tracking code uses each -progress metric separately to come up with an overall completion -percentage and an ETA value. - - <p>To connect this parser into the <code>Trial</code> BuildStep, -<code>Trial.__init__</code> ends with the following clause: - -<pre class="example"> # this counter will feed Progress along the 'test cases' metric - counter = TrialTestCaseCounter() - self.addLogObserver('stdio', counter) - self.progressMetrics += ('tests',) -</pre> - <p>This creates a TrialTestCaseCounter and tells the step that the -counter wants to watch the “stdio” log. The observer is -automatically given a reference to the step in its <code>.step</code> -attribute. - -<h4 class="subheading">A Somewhat Whimsical Example</h4> - -<p>Let's say that we've got some snazzy new unit-test framework called -Framboozle. It's the hottest thing since sliced bread. It slices, it -dices, it runs unit tests like there's no tomorrow. Plus if your unit -tests fail, you can use its name for a Web 2.1 startup company, make -millions of dollars, and hire engineers to fix the bugs for you, while -you spend your afternoons lazily hang-gliding along a scenic pacific -beach, blissfully unconcerned about the state of your -tests.<a rel="footnote" href="#fn-8" name="fnd-8"><sup>8</sup></a> - - <p>To run a Framboozle-enabled test suite, you just run the 'framboozler' -command from the top of your source code tree. The 'framboozler' -command emits a bunch of stuff to stdout, but the most interesting bit -is that it emits the line "FNURRRGH!" every time it finishes running a -test case<a rel="footnote" href="#fn-9" name="fnd-9"><sup>9</sup></a>. You'd like to have a test-case counting LogObserver that -watches for these lines and counts them, because counting them will -help the buildbot more accurately calculate how long the build will -take, and this will let you know exactly how long you can sneak out of -the office for your hang-gliding lessons without anyone noticing that -you're gone. - - <p>This will involve writing a new BuildStep (probably named -"Framboozle") which inherits from ShellCommand. The BuildStep class -definition itself will look something like this: - -<pre class="example"> # START - from buildbot.steps.shell import ShellCommand - from buildbot.process.buildstep import LogLineObserver - - class FNURRRGHCounter(LogLineObserver): - numTests = 0 - def outLineReceived(self, line): - if "FNURRRGH!" in line: - self.numTests += 1 - self.step.setProgress('tests', self.numTests) - - class Framboozle(ShellCommand): - command = ["framboozler"] - - def __init__(self, **kwargs): - ShellCommand.__init__(self, **kwargs) # always upcall! - counter = FNURRRGHCounter()) - self.addLogObserver('stdio', counter) - self.progressMetrics += ('tests',) - # FINISH -</pre> - <p>So that's the code that we want to wind up using. How do we actually -deploy it? - - <p>You have a couple of different options. - - <p>Option 1: The simplest technique is to simply put this text -(everything from START to FINISH) in your master.cfg file, somewhere -before the BuildFactory definition where you actually use it in a -clause like: - -<pre class="example"> f = BuildFactory() - f.addStep(SVN(svnurl="stuff")) - f.addStep(Framboozle()) -</pre> - <p>Remember that master.cfg is secretly just a python program with one -job: populating the BuildmasterConfig dictionary. And python programs -are allowed to define as many classes as they like. So you can define -classes and use them in the same file, just as long as the class is -defined before some other code tries to use it. - - <p>This is easy, and it keeps the point of definition very close to the -point of use, and whoever replaces you after that unfortunate -hang-gliding accident will appreciate being able to easily figure out -what the heck this stupid "Framboozle" step is doing anyways. The -downside is that every time you reload the config file, the Framboozle -class will get redefined, which means that the buildmaster will think -that you've reconfigured all the Builders that use it, even though -nothing changed. Bleh. - - <p>Option 2: Instead, we can put this code in a separate file, and import -it into the master.cfg file just like we would the normal buildsteps -like ShellCommand and SVN. - - <p>Create a directory named ~/lib/python, put everything from START to -FINISH in ~/lib/python/framboozle.py, and run your buildmaster using: - -<pre class="example"> PYTHONPATH=~/lib/python buildbot start MASTERDIR -</pre> - <p>or use the <samp><span class="file">Makefile.buildbot</span></samp> to control the way -<samp><span class="command">buildbot start</span></samp> works. Or add something like this to -something like your ~/.bashrc or ~/.bash_profile or ~/.cshrc: - -<pre class="example"> export PYTHONPATH=~/lib/python -</pre> - <p>Once we've done this, our master.cfg can look like: - -<pre class="example"> from framboozle import Framboozle - f = BuildFactory() - f.addStep(SVN(svnurl="stuff")) - f.addStep(Framboozle()) -</pre> - <p>or: - -<pre class="example"> import framboozle - f = BuildFactory() - f.addStep(SVN(svnurl="stuff")) - f.addStep(framboozle.Framboozle()) -</pre> - <p>(check out the python docs for details about how "import" and "from A -import B" work). - - <p>What we've done here is to tell python that every time it handles an -"import" statement for some named module, it should look in our -~/lib/python/ for that module before it looks anywhere else. After our -directories, it will try in a bunch of standard directories too -(including the one where buildbot is installed). By setting the -PYTHONPATH environment variable, you can add directories to the front -of this search list. - - <p>Python knows that once it "import"s a file, it doesn't need to -re-import it again. This means that reconfiguring the buildmaster -(with "buildbot reconfig", for example) won't make it think the -Framboozle class has changed every time, so the Builders that use it -will not be spuriously restarted. On the other hand, you either have -to start your buildmaster in a slightly weird way, or you have to -modify your environment to set the PYTHONPATH variable. - - <p>Option 3: Install this code into a standard python library directory - - <p>Find out what your python's standard include path is by asking it: - -<pre class="example"> 80:warner@luther% python - Python 2.4.4c0 (#2, Oct 2 2006, 00:57:46) - [GCC 4.1.2 20060928 (prerelease) (Debian 4.1.1-15)] on linux2 - Type "help", "copyright", "credits" or "license" for more information. - >>> import sys - >>> import pprint - >>> pprint.pprint(sys.path) - ['', - '/usr/lib/python24.zip', - '/usr/lib/python2.4', - '/usr/lib/python2.4/plat-linux2', - '/usr/lib/python2.4/lib-tk', - '/usr/lib/python2.4/lib-dynload', - '/usr/local/lib/python2.4/site-packages', - '/usr/lib/python2.4/site-packages', - '/usr/lib/python2.4/site-packages/Numeric', - '/var/lib/python-support/python2.4', - '/usr/lib/site-python'] -</pre> - <p>In this case, putting the code into -/usr/local/lib/python2.4/site-packages/framboozle.py would work just -fine. We can use the same master.cfg "import framboozle" statement as -in Option 2. By putting it in a standard include directory (instead of -the decidedly non-standard ~/lib/python), we don't even have to set -PYTHONPATH to anything special. The downside is that you probably have -to be root to write to one of those standard include directories. - - <p>Option 4: Submit the code for inclusion in the Buildbot distribution - - <p>Make a fork of buildbot on http://github.com/djmitche/buildbot or post a patch -in a bug at http://buildbot.net. In either case, post a note about your patch -to the mailing list, so others can provide feedback and, eventually, commit it. - -<pre class="example"> from buildbot.steps import framboozle - f = BuildFactory() - f.addStep(SVN(svnurl="stuff")) - f.addStep(framboozle.Framboozle()) -</pre> - <p>And then you don't even have to install framboozle.py anywhere on your -system, since it will ship with Buildbot. You don't have to be root, -you don't have to set PYTHONPATH. But you do have to make a good case -for Framboozle being worth going into the main distribution, you'll -probably have to provide docs and some unit test cases, you'll need to -figure out what kind of beer the author likes, and then you'll have to -wait until the next release. But in some environments, all this is -easier than getting root on your buildmaster box, so the tradeoffs may -actually be worth it. - - <p>Putting the code in master.cfg (1) makes it available to that -buildmaster instance. Putting it in a file in a personal library -directory (2) makes it available for any buildmasters you might be -running. Putting it in a file in a system-wide shared library -directory (3) makes it available for any buildmasters that anyone on -that system might be running. Getting it into the buildbot's upstream -repository (4) makes it available for any buildmasters that anyone in -the world might be running. It's all a matter of how widely you want -to deploy that new class. - -<div class="node"> -<p><hr> -<a name="BuildStep-URLs"></a> -Previous: <a rel="previous" accesskey="p" href="#Adding-LogObservers">Adding LogObservers</a>, -Up: <a rel="up" accesskey="u" href="#Writing-New-BuildSteps">Writing New BuildSteps</a> - -</div> - -<h5 class="subsubsection">6.1.10.5 BuildStep URLs</h5> - -<p><a name="index-links-87"></a><a name="index-BuildStep-URLs-88"></a><a name="index-addURL-89"></a> -Each BuildStep has a collection of “links”. Like its collection of -LogFiles, each link has a name and a target URL. The web status page -creates HREFs for each link in the same box as it does for LogFiles, -except that the target of the link is the external URL instead of an -internal link to a page that shows the contents of the LogFile. - - <p>These external links can be used to point at build information hosted -on other servers. For example, the test process might produce an -intricate description of which tests passed and failed, or some sort -of code coverage data in HTML form, or a PNG or GIF image with a graph -of memory usage over time. The external link can provide an easy way -for users to navigate from the buildbot's status page to these -external web sites or file servers. Note that the step itself is -responsible for insuring that there will be a document available at -the given URL (perhaps by using <samp><span class="command">scp</span></samp> to copy the HTML output -to a <samp><span class="file">~/public_html/</span></samp> directory on a remote web server). Calling -<code>addURL</code> does not magically populate a web server. - - <p>To set one of these links, the BuildStep should call the <code>addURL</code> -method with the name of the link and the target URL. Multiple URLs can -be set. - - <p>In this example, we assume that the <samp><span class="command">make test</span></samp> command causes -a collection of HTML files to be created and put somewhere on the -coverage.example.org web server, in a filename that incorporates the -build number. - -<pre class="example"> class TestWithCodeCoverage(BuildStep): - command = ["make", "test", - WithProperties("buildnum=%s" % "buildnumber")] - - def createSummary(self, log): - buildnumber = self.getProperty("buildnumber") - url = "http://coverage.example.org/builds/%s.html" % buildnumber - self.addURL("coverage", url) -</pre> - <p>You might also want to extract the URL from some special message -output by the build process itself: - -<pre class="example"> class TestWithCodeCoverage(BuildStep): - command = ["make", "test", - WithProperties("buildnum=%s" % "buildnumber")] - - def createSummary(self, log): - output = StringIO(log.getText()) - for line in output.readlines(): - if line.startswith("coverage-url:"): - url = line[len("coverage-url:"):].strip() - self.addURL("coverage", url) - return -</pre> - <p>Note that a build process which emits both stdout and stderr might -cause this line to be split or interleaved between other lines. It -might be necessary to restrict the getText() call to only stdout with -something like this: - -<pre class="example"> output = StringIO("".join([c[1] - for c in log.getChunks() - if c[0] == LOG_CHANNEL_STDOUT])) -</pre> - <p>Of course if the build is run under a PTY, then stdout and stderr will -be merged before the buildbot ever sees them, so such interleaving -will be unavoidable. - -<div class="node"> -<p><hr> -<a name="Interlocks"></a> -Next: <a rel="next" accesskey="n" href="#Build-Factories">Build Factories</a>, -Previous: <a rel="previous" accesskey="p" href="#Build-Steps">Build Steps</a>, -Up: <a rel="up" accesskey="u" href="#Build-Process">Build Process</a> - -</div> - -<h3 class="section">6.2 Interlocks</h3> - -<p><a name="index-locks-90"></a><a name="index-buildbot_002elocks_002eMasterLock-91"></a><a name="index-buildbot_002elocks_002eSlaveLock-92"></a><a name="index-buildbot_002elocks_002eLockAccess-93"></a> -Until now, we assumed that a master can run builds at any slave whenever -needed or desired. Some times, you want to enforce additional constraints on -builds. For reasons like limited network bandwidth, old slave machines, or a -self-willed data base server, you may want to limit the number of builds (or -build steps) that can access a resource. - - <p>The mechanism used by Buildbot is known as the read/write lock.<a rel="footnote" href="#fn-10" name="fnd-10"><sup>10</sup></a> It -allows either many readers or a single writer but not a combination of readers -and writers. The general lock has been modified and extended for use in -Buildbot. Firstly, the general lock allows an infinite number of readers. In -Buildbot, we often want to put an upper limit on the number of readers, for -example allowing two out of five possible builds at the same time. To do this, -the lock counts the number of active readers. Secondly, the terms <em>read -mode</em> and <em>write mode</em> are confusing in Buildbot context. They have been -replaced by <em>counting mode</em> (since the lock counts them) and <em>exclusive -mode</em>. As a result of these changes, locks in Buildbot allow a number of -builds (upto some fixed number) in counting mode, or they allow one build in -exclusive mode. - - <p>Often, not all slaves are equal. To allow for this situation, Buildbot allows -to have a separate upper limit on the count for each slave. In this way, you -can have at most 3 concurrent builds at a fast slave, 2 at a slightly older -slave, and 1 at all other slaves. - - <p>The final thing you can specify when you introduce a new lock is its scope. -Some constraints are global – they must be enforced over all slaves. Other -constraints are local to each slave. A <em>master lock</em> is used for the -global constraints. You can ensure for example that at most one build (of all -builds running at all slaves) accesses the data base server. With a -<em>slave lock</em> you can add a limit local to each slave. With such a lock, -you can for example enforce an upper limit to the number of active builds at a -slave, like above. - - <p>Time for a few examples. Below a master lock is defined to protect a data base, -and a slave lock is created to limit the number of builds at each slave. - -<pre class="example"> from buildbot import locks - - db_lock = locks.MasterLock("database") - build_lock = locks.SlaveLock("slave_builds", - maxCount = 1, - maxCountForSlave = { 'fast': 3, 'new': 2 }) -</pre> - <p>After importing locks from buildbot, <code>db_lock</code> is defined to be a master -lock. The <code>"database"</code> string is used for uniquely identifying the lock. -At the next line, a slave lock called <code>build_lock</code> is created. It is -identified by the <code>"slave_builds"</code> string. Since the requirements of the -lock are a bit more complicated, two optional arguments are also specified. The -<code>maxCount</code> parameter sets the default limit for builds in counting mode to -<code>1</code>. For the slave called <code>'fast'</code> however, we want to have at most -three builds, and for the slave called <code>'new'</code> the upper limit is two -builds running at the same time. - - <p>The next step is using the locks in builds. Buildbot allows a lock to be used -during an entire build (from beginning to end), or only during a single build -step. In the latter case, the lock is claimed for use just before the step -starts, and released again when the step ends. To prevent -deadlocks,<a rel="footnote" href="#fn-11" name="fnd-11"><sup>11</sup></a> it is not possible to claim or release -locks at other times. - - <p>To use locks, you should add them with a <code>locks</code> argument. -Each use of a lock is either in counting mode (that is, possibly shared with -other builds) or in exclusive mode. A build or build step proceeds only when it -has acquired all locks. If a build or step needs a lot of locks, it may be -starved<a rel="footnote" href="#fn-12" name="fnd-12"><sup>12</sup></a> by other builds that need fewer locks. - - <p>To illustrate use of locks, a few examples. - -<pre class="example"> from buildbot import locks - from buildbot.steps import source, shell - from buildbot.process import factory - - db_lock = locks.MasterLock("database") - build_lock = locks.SlaveLock("slave_builds", - maxCount = 1, - maxCountForSlave = { 'fast': 3, 'new': 2 }) - - f = factory.BuildFactory() - f.addStep(source.SVN(svnurl="http://example.org/svn/Trunk")) - f.addStep(shell.ShellCommand(command="make all")) - f.addStep(shell.ShellCommand(command="make test", - locks=[db_lock.access('exclusive')])) - - b1 = {'name': 'full1', 'slavename': 'fast', 'builddir': 'f1', 'factory': f, - 'locks': [build_lock.access('counting')] } - - b2 = {'name': 'full2', 'slavename': 'new', 'builddir': 'f2', 'factory': f. - 'locks': [build_lock.access('counting')] } - - b3 = {'name': 'full3', 'slavename': 'old', 'builddir': 'f3', 'factory': f. - 'locks': [build_lock.access('counting')] } - - b4 = {'name': 'full4', 'slavename': 'other', 'builddir': 'f4', 'factory': f. - 'locks': [build_lock.access('counting')] } - - c['builders'] = [b1, b2, b3, b4] -</pre> - <p>Here we have four slaves <code>b1</code>, <code>b2</code>, <code>b3</code>, and <code>b4</code>. Each -slave performs the same checkout, make, and test build step sequence. -We want to enforce that at most one test step is executed between all slaves due -to restrictions with the data base server. This is done by adding the -<code>locks=</code> parameter with the third step. It takes a list of locks with their -access mode. In this case only the <code>db_lock</code> is needed. The exclusive -access mode is used to ensure there is at most one slave that executes the test -step. - - <p>In addition to exclusive accessing the data base, we also want slaves to stay -responsive even under the load of a large number of builds being triggered. -For this purpose, the slave lock called <code>build_lock</code> is defined. Since -the restraint holds for entire builds, the lock is specified in the builder -with <code>'locks': [build_lock.access('counting')]</code>. -<div class="node"> -<p><hr> -<a name="Build-Factories"></a> -Previous: <a rel="previous" accesskey="p" href="#Interlocks">Interlocks</a>, -Up: <a rel="up" accesskey="u" href="#Build-Process">Build Process</a> - -</div> - -<h3 class="section">6.3 Build Factories</h3> - -<p>Each Builder is equipped with a “build factory”, which is -responsible for producing the actual <code>Build</code> objects that perform -each build. This factory is created in the configuration file, and -attached to a Builder through the <code>factory</code> element of its -dictionary. - - <p>The standard <code>BuildFactory</code> object creates <code>Build</code> objects -by default. These Builds will each execute a collection of BuildSteps -in a fixed sequence. Each step can affect the results of the build, -but in general there is little intelligence to tie the different steps -together. You can create subclasses of <code>Build</code> to implement more -sophisticated build processes, and then use a subclass of -<code>BuildFactory</code> (or simply set the <code>buildClass</code> attribute) to -create instances of your new Build subclass. - -<ul class="menu"> -<li><a accesskey="1" href="#BuildStep-Objects">BuildStep Objects</a> -<li><a accesskey="2" href="#BuildFactory">BuildFactory</a> -<li><a accesskey="3" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a> -</ul> - -<div class="node"> -<p><hr> -<a name="BuildStep-Objects"></a> -Next: <a rel="next" accesskey="n" href="#BuildFactory">BuildFactory</a>, -Previous: <a rel="previous" accesskey="p" href="#Build-Factories">Build Factories</a>, -Up: <a rel="up" accesskey="u" href="#Build-Factories">Build Factories</a> - -</div> - -<h4 class="subsection">6.3.1 BuildStep Objects</h4> - -<p>The steps used by these builds are all subclasses of <code>BuildStep</code>. -The standard ones provided with Buildbot are documented later, -See <a href="#Build-Steps">Build Steps</a>. You can also write your own subclasses to use in -builds. - - <p>The basic behavior for a <code>BuildStep</code> is to: - - <ul> -<li>run for a while, then stop -<li>possibly invoke some RemoteCommands on the attached build slave -<li>possibly produce a set of log files -<li>finish with a status described by one of four values defined in -buildbot.status.builder: SUCCESS, WARNINGS, FAILURE, SKIPPED -<li>provide a list of short strings to describe the step -<li>define a color (generally green, orange, or red) with which the -step should be displayed -</ul> - - <p>More sophisticated steps may produce additional information and -provide it to later build steps, or store it in the factory to provide -to later builds. - -<ul class="menu"> -<li><a accesskey="1" href="#BuildFactory-Attributes">BuildFactory Attributes</a> -<li><a accesskey="2" href="#Quick-builds">Quick builds</a> -</ul> - -<div class="node"> -<p><hr> -<a name="BuildFactory"></a> -Next: <a rel="next" accesskey="n" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>, -Previous: <a rel="previous" accesskey="p" href="#BuildStep-Objects">BuildStep Objects</a>, -Up: <a rel="up" accesskey="u" href="#Build-Factories">Build Factories</a> - -</div> - -<h4 class="subsection">6.3.2 BuildFactory</h4> - -<p><a name="index-buildbot_002eprocess_002efactory_002eBuildFactory-94"></a><a name="index-buildbot_002eprocess_002efactory_002eBasicBuildFactory-95"></a><!-- TODO: what is BasicSVN anyway? --> -<a name="index-buildbot_002eprocess_002efactory_002eBasicSVN-96"></a> -The default <code>BuildFactory</code>, provided in the -<code>buildbot.process.factory</code> module, contains an internal list of -“BuildStep specifications”: a list of <code>(step_class, kwargs)</code> -tuples for each. These specification tuples are constructed when the -config file is read, by asking the instances passed to <code>addStep</code> -for their subclass and arguments. - - <p>When asked to create a Build, the <code>BuildFactory</code> puts a copy of -the list of step specifications into the new Build object. When the -Build is actually started, these step specifications are used to -create the actual set of BuildSteps, which are then executed one at a -time. This serves to give each Build an independent copy of each step. -For example, a build which consists of a CVS checkout followed by a -<code>make build</code> would be constructed as follows: - -<pre class="example"> from buildbot.steps import source, shell - from buildbot.process import factory - - f = factory.BuildFactory() - f.addStep(source.CVS(cvsroot=CVSROOT, cvsmodule="project", mode="update")) - f.addStep(shell.Compile(command=["make", "build"])) -</pre> - <p>(To support config files from buildbot-0.7.5 and earlier, -<code>addStep</code> also accepts the <code>f.addStep(shell.Compile, -command=["make","build"])</code> form, although its use is discouraged -because then the <code>Compile</code> step doesn't get to validate or -complain about its arguments until build time. The modern -pass-by-instance approach allows this validation to occur while the -config file is being loaded, where the admin has a better chance of -noticing problems). - - <p>It is also possible to pass a list of steps into the -<code>BuildFactory</code> when it is created. Using <code>addStep</code> is -usually simpler, but there are cases where is is more convenient to -create the list of steps ahead of time.: - -<pre class="example"> from buildbot.steps import source, shell - from buildbot.process import factory - - all_steps = [source.CVS(cvsroot=CVSROOT, cvsmodule="project", mode="update"), - shell.Compile(command=["make", "build"]), - ] - f = factory.BuildFactory(all_steps) -</pre> - <p>Each step can affect the build process in the following ways: - - <ul> -<li>If the step's <code>haltOnFailure</code> attribute is True, then a failure -in the step (i.e. if it completes with a result of FAILURE) will cause -the whole build to be terminated immediately: no further steps will be -executed, with the exception of steps with <code>alwaysRun</code> set to -True. <code>haltOnFailure</code> is useful for setup steps upon which the -rest of the build depends: if the CVS checkout or <code>./configure</code> -process fails, there is no point in trying to compile or test the -resulting tree. - - <li>If the step's <code>alwaysRun</code> attribute is True, then it will always -be run, regardless of if previous steps have failed. This is useful -for cleanup steps that should always be run to return the build -directory or build slave into a good state. - - <li>If the <code>flunkOnFailure</code> or <code>flunkOnWarnings</code> flag is set, -then a result of FAILURE or WARNINGS will mark the build as a whole as -FAILED. However, the remaining steps will still be executed. This is -appropriate for things like multiple testing steps: a failure in any -one of them will indicate that the build has failed, however it is -still useful to run them all to completion. - - <li>Similarly, if the <code>warnOnFailure</code> or <code>warnOnWarnings</code> flag -is set, then a result of FAILURE or WARNINGS will mark the build as -having WARNINGS, and the remaining steps will still be executed. This -may be appropriate for certain kinds of optional build or test steps. -For example, a failure experienced while building documentation files -should be made visible with a WARNINGS result but not be serious -enough to warrant marking the whole build with a FAILURE. - - </ul> - - <p>In addition, each Step produces its own results, may create logfiles, -etc. However only the flags described above have any effect on the -build as a whole. - - <p>The pre-defined BuildSteps like <code>CVS</code> and <code>Compile</code> have -reasonably appropriate flags set on them already. For example, without -a source tree there is no point in continuing the build, so the -<code>CVS</code> class has the <code>haltOnFailure</code> flag set to True. Look -in <samp><span class="file">buildbot/steps/*.py</span></samp> to see how the other Steps are -marked. - - <p>Each Step is created with an additional <code>workdir</code> argument that -indicates where its actions should take place. This is specified as a -subdirectory of the slave builder's base directory, with a default -value of <code>build</code>. This is only implemented as a step argument (as -opposed to simply being a part of the base directory) because the -CVS/SVN steps need to perform their checkouts from the parent -directory. - -<ul class="menu"> -<li><a accesskey="1" href="#BuildFactory-Attributes">BuildFactory Attributes</a> -<li><a accesskey="2" href="#Quick-builds">Quick builds</a> -</ul> - -<div class="node"> -<p><hr> -<a name="BuildFactory-Attributes"></a> -Next: <a rel="next" accesskey="n" href="#Quick-builds">Quick builds</a>, -Previous: <a rel="previous" accesskey="p" href="#BuildFactory">BuildFactory</a>, -Up: <a rel="up" accesskey="u" href="#BuildFactory">BuildFactory</a> - -</div> - -<h5 class="subsubsection">6.3.2.1 BuildFactory Attributes</h5> - -<p>Some attributes from the BuildFactory are copied into each Build. - - <p><a name="index-treeStableTimer-97"></a> - <dl> -<dt><code>useProgress</code><dd>(defaults to True): if True, the buildmaster keeps track of how long -each step takes, so it can provide estimates of how long future builds -will take. If builds are not expected to take a consistent amount of -time (such as incremental builds in which a random set of files are -recompiled or tested each time), this should be set to False to -inhibit progress-tracking. - - </dl> - -<div class="node"> -<p><hr> -<a name="Quick-builds"></a> -Previous: <a rel="previous" accesskey="p" href="#BuildFactory-Attributes">BuildFactory Attributes</a>, -Up: <a rel="up" accesskey="u" href="#BuildFactory">BuildFactory</a> - -</div> - -<h5 class="subsubsection">6.3.2.2 Quick builds</h5> - -<p><a name="index-buildbot_002eprocess_002efactory_002eQuickBuildFactory-98"></a> -The difference between a “full build” and a “quick build” is that -quick builds are generally done incrementally, starting with the tree -where the previous build was performed. That simply means that the -source-checkout step should be given a <code>mode='update'</code> flag, to -do the source update in-place. - - <p>In addition to that, the <code>useProgress</code> flag should be set to -False. Incremental builds will (or at least the ought to) compile as -few files as necessary, so they will take an unpredictable amount of -time to run. Therefore it would be misleading to claim to predict how -long the build will take. - -<div class="node"> -<p><hr> -<a name="Process-Specific-build-factories"></a> -<a name="Process_002dSpecific-build-factories"></a> -Previous: <a rel="previous" accesskey="p" href="#BuildFactory">BuildFactory</a>, -Up: <a rel="up" accesskey="u" href="#Build-Factories">Build Factories</a> - -</div> - -<h4 class="subsection">6.3.3 Process-Specific build factories</h4> - -<p>Many projects use one of a few popular build frameworks to simplify -the creation and maintenance of Makefiles or other compilation -structures. Buildbot provides several pre-configured BuildFactory -subclasses which let you build these projects with a minimum of fuss. - -<ul class="menu"> -<li><a accesskey="1" href="#GNUAutoconf">GNUAutoconf</a> -<li><a accesskey="2" href="#CPAN">CPAN</a> -<li><a accesskey="3" href="#Python-distutils">Python distutils</a> -<li><a accesskey="4" href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a> -</ul> - -<div class="node"> -<p><hr> -<a name="GNUAutoconf"></a> -Next: <a rel="next" accesskey="n" href="#CPAN">CPAN</a>, -Previous: <a rel="previous" accesskey="p" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>, -Up: <a rel="up" accesskey="u" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a> - -</div> - -<h5 class="subsubsection">6.3.3.1 GNUAutoconf</h5> - -<p><a name="index-buildbot_002eprocess_002efactory_002eGNUAutoconf-99"></a> -<a href="http://www.gnu.org/software/autoconf/">GNU Autoconf</a> is a -software portability tool, intended to make it possible to write -programs in C (and other languages) which will run on a variety of -UNIX-like systems. Most GNU software is built using autoconf. It is -frequently used in combination with GNU automake. These tools both -encourage a build process which usually looks like this: - -<pre class="example"> % CONFIG_ENV=foo ./configure --with-flags - % make all - % make check - # make install -</pre> - <p>(except of course the Buildbot always skips the <code>make install</code> -part). - - <p>The Buildbot's <code>buildbot.process.factory.GNUAutoconf</code> factory is -designed to build projects which use GNU autoconf and/or automake. The -configuration environment variables, the configure flags, and command -lines used for the compile and test are all configurable, in general -the default values will be suitable. - - <p>Example: - -<pre class="example"> # use the s() convenience function defined earlier - f = factory.GNUAutoconf(source=s(step.SVN, svnurl=URL, mode="copy"), - flags=["--disable-nls"]) -</pre> - <p>Required Arguments: - - <dl> -<dt><code>source</code><dd>This argument must be a step specification tuple that provides a -BuildStep to generate the source tree. -</dl> - - <p>Optional Arguments: - - <dl> -<dt><code>configure</code><dd>The command used to configure the tree. Defaults to -<code>./configure</code>. Accepts either a string or a list of shell argv -elements. - - <br><dt><code>configureEnv</code><dd>The environment used for the initial configuration step. This accepts -a dictionary which will be merged into the buildslave's normal -environment. This is commonly used to provide things like -<code>CFLAGS="-O2 -g"</code> (to turn off debug symbols during the compile). -Defaults to an empty dictionary. - - <br><dt><code>configureFlags</code><dd>A list of flags to be appended to the argument list of the configure -command. This is commonly used to enable or disable specific features -of the autoconf-controlled package, like <code>["--without-x"]</code> to -disable windowing support. Defaults to an empty list. - - <br><dt><code>compile</code><dd>this is a shell command or list of argv values which is used to -actually compile the tree. It defaults to <code>make all</code>. If set to -None, the compile step is skipped. - - <br><dt><code>test</code><dd>this is a shell command or list of argv values which is used to run -the tree's self-tests. It defaults to <code>make check</code>. If set to -None, the test step is skipped. - - </dl> - -<div class="node"> -<p><hr> -<a name="CPAN"></a> -Next: <a rel="next" accesskey="n" href="#Python-distutils">Python distutils</a>, -Previous: <a rel="previous" accesskey="p" href="#GNUAutoconf">GNUAutoconf</a>, -Up: <a rel="up" accesskey="u" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a> - -</div> - -<h5 class="subsubsection">6.3.3.2 CPAN</h5> - -<p><a name="index-buildbot_002eprocess_002efactory_002eCPAN-100"></a> -Most Perl modules available from the <a href="http://www.cpan.org/">CPAN</a> -archive use the <code>MakeMaker</code> module to provide configuration, -build, and test services. The standard build routine for these modules -looks like: - -<pre class="example"> % perl Makefile.PL - % make - % make test - # make install -</pre> - <p>(except again Buildbot skips the install step) - - <p>Buildbot provides a <code>CPAN</code> factory to compile and test these -projects. - - <p>Arguments: - <dl> -<dt><code>source</code><dd>(required): A step specification tuple, like that used by GNUAutoconf. - - <br><dt><code>perl</code><dd>A string which specifies the <code>perl</code> executable to use. Defaults -to just <code>perl</code>. - - </dl> - -<div class="node"> -<p><hr> -<a name="Python-distutils"></a> -Next: <a rel="next" accesskey="n" href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a>, -Previous: <a rel="previous" accesskey="p" href="#CPAN">CPAN</a>, -Up: <a rel="up" accesskey="u" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a> - -</div> - -<h5 class="subsubsection">6.3.3.3 Python distutils</h5> - -<p><a name="index-buildbot_002eprocess_002efactory_002eDistutils-101"></a> -Most Python modules use the <code>distutils</code> package to provide -configuration and build services. The standard build process looks -like: - -<pre class="example"> % python ./setup.py build - % python ./setup.py install -</pre> - <p>Unfortunately, although Python provides a standard unit-test framework -named <code>unittest</code>, to the best of my knowledge <code>distutils</code> -does not provide a standardized target to run such unit tests. (Please -let me know if I'm wrong, and I will update this factory.) - - <p>The <code>Distutils</code> factory provides support for running the build -part of this process. It accepts the same <code>source=</code> parameter as -the other build factories. - - <p>Arguments: - <dl> -<dt><code>source</code><dd>(required): A step specification tuple, like that used by GNUAutoconf. - - <br><dt><code>python</code><dd>A string which specifies the <code>python</code> executable to use. Defaults -to just <code>python</code>. - - <br><dt><code>test</code><dd>Provides a shell command which runs unit tests. This accepts either a -string or a list. The default value is None, which disables the test -step (since there is no common default command to run unit tests in -distutils modules). - - </dl> - -<div class="node"> -<p><hr> -<a name="Python%2fTwisted%2ftrial-projects"></a> -<a name="Python_002fTwisted_002ftrial-projects"></a> -Previous: <a rel="previous" accesskey="p" href="#Python-distutils">Python distutils</a>, -Up: <a rel="up" accesskey="u" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a> - -</div> - -<h5 class="subsubsection">6.3.3.4 Python/Twisted/trial projects</h5> - -<p><a name="index-buildbot_002eprocess_002efactory_002eTrial-102"></a><!-- TODO: document these steps better --> -<a name="index-buildbot_002esteps_002epython_005ftwisted_002eHLint-103"></a><a name="index-buildbot_002esteps_002epython_005ftwisted_002eTrial-104"></a><a name="index-buildbot_002esteps_002epython_005ftwisted_002eProcessDocs-105"></a><a name="index-buildbot_002esteps_002epython_005ftwisted_002eBuildDebs-106"></a><a name="index-buildbot_002esteps_002epython_005ftwisted_002eRemovePYCs-107"></a> -Twisted provides a unit test tool named <code>trial</code> which provides a -few improvements over Python's built-in <code>unittest</code> module. Many -python projects which use Twisted for their networking or application -services also use trial for their unit tests. These modules are -usually built and tested with something like the following: - -<pre class="example"> % python ./setup.py build - % PYTHONPATH=build/lib.linux-i686-2.3 trial -v PROJECTNAME.test - % python ./setup.py install -</pre> - <p>Unfortunately, the <samp><span class="file">build/lib</span></samp> directory into which the -built/copied .py files are placed is actually architecture-dependent, -and I do not yet know of a simple way to calculate its value. For many -projects it is sufficient to import their libraries “in place” from -the tree's base directory (<code>PYTHONPATH=.</code>). - - <p>In addition, the <var>PROJECTNAME</var> value where the test files are -located is project-dependent: it is usually just the project's -top-level library directory, as common practice suggests the unit test -files are put in the <code>test</code> sub-module. This value cannot be -guessed, the <code>Trial</code> class must be told where to find the test -files. - - <p>The <code>Trial</code> class provides support for building and testing -projects which use distutils and trial. If the test module name is -specified, trial will be invoked. The library path used for testing -can also be set. - - <p>One advantage of trial is that the Buildbot happens to know how to -parse trial output, letting it identify which tests passed and which -ones failed. The Buildbot can then provide fine-grained reports about -how many tests have failed, when individual tests fail when they had -been passing previously, etc. - - <p>Another feature of trial is that you can give it a series of source -.py files, and it will search them for special <code>test-case-name</code> -tags that indicate which test cases provide coverage for that file. -Trial can then run just the appropriate tests. This is useful for -quick builds, where you want to only run the test cases that cover the -changed functionality. - - <p>Arguments: - <dl> -<dt><code>source</code><dd>(required): A step specification tuple, like that used by GNUAutoconf. - - <br><dt><code>buildpython</code><dd>A list (argv array) of strings which specifies the <code>python</code> -executable to use when building the package. Defaults to just -<code>['python']</code>. It may be useful to add flags here, to supress -warnings during compilation of extension modules. This list is -extended with <code>['./setup.py', 'build']</code> and then executed in a -ShellCommand. - - <br><dt><code>testpath</code><dd>Provides a directory to add to <code>PYTHONPATH</code> when running the unit -tests, if tests are being run. Defaults to <code>.</code> to include the -project files in-place. The generated build library is frequently -architecture-dependent, but may simply be <samp><span class="file">build/lib</span></samp> for -pure-python modules. - - <br><dt><code>trialpython</code><dd>Another list of strings used to build the command that actually runs -trial. This is prepended to the contents of the <code>trial</code> argument -below. It may be useful to add <code>-W</code> flags here to supress -warnings that occur while tests are being run. Defaults to an empty -list, meaning <code>trial</code> will be run without an explicit -interpreter, which is generally what you want if you're using -<samp><span class="file">/usr/bin/trial</span></samp> instead of, say, the <samp><span class="file">./bin/trial</span></samp> that -lives in the Twisted source tree. - - <br><dt><code>trial</code><dd>provides the name of the <code>trial</code> command. It is occasionally -useful to use an alternate executable, such as <code>trial2.2</code> which -might run the tests under an older version of Python. Defaults to -<code>trial</code>. - - <br><dt><code>tests</code><dd>Provides a module name or names which contain the unit tests for this -project. Accepts a string, typically <code>PROJECTNAME.test</code>, or a -list of strings. Defaults to None, indicating that no tests should be -run. You must either set this or <code>useTestCaseNames</code> to do anyting -useful with the Trial factory. - - <br><dt><code>useTestCaseNames</code><dd>Tells the Step to provide the names of all changed .py files to trial, -so it can look for test-case-name tags and run just the matching test -cases. Suitable for use in quick builds. Defaults to False. - - <br><dt><code>randomly</code><dd>If <code>True</code>, tells Trial (with the <code>--random=0</code> argument) to -run the test cases in random order, which sometimes catches subtle -inter-test dependency bugs. Defaults to <code>False</code>. - - <br><dt><code>recurse</code><dd>If <code>True</code>, tells Trial (with the <code>--recurse</code> argument) to -look in all subdirectories for additional test cases. It isn't clear -to me how this works, but it may be useful to deal with the -unknown-PROJECTNAME problem described above, and is currently used in -the Twisted buildbot to accomodate the fact that test cases are now -distributed through multiple twisted.SUBPROJECT.test directories. - - </dl> - - <p>Unless one of <code>trialModule</code> or <code>useTestCaseNames</code> -are set, no tests will be run. - - <p>Some quick examples follow. Most of these examples assume that the -target python code (the “code under test”) can be reached directly -from the root of the target tree, rather than being in a <samp><span class="file">lib/</span></samp> -subdirectory. - -<pre class="example"> # Trial(source, tests="toplevel.test") does: - # python ./setup.py build - # PYTHONPATH=. trial -to toplevel.test - - # Trial(source, tests=["toplevel.test", "other.test"]) does: - # python ./setup.py build - # PYTHONPATH=. trial -to toplevel.test other.test - - # Trial(source, useTestCaseNames=True) does: - # python ./setup.py build - # PYTHONPATH=. trial -to --testmodule=foo/bar.py.. (from Changes) - - # Trial(source, buildpython=["python2.3", "-Wall"], tests="foo.tests"): - # python2.3 -Wall ./setup.py build - # PYTHONPATH=. trial -to foo.tests - - # Trial(source, trialpython="python2.3", trial="/usr/bin/trial", - # tests="foo.tests") does: - # python2.3 -Wall ./setup.py build - # PYTHONPATH=. python2.3 /usr/bin/trial -to foo.tests - - # For running trial out of the tree being tested (only useful when the - # tree being built is Twisted itself): - # Trial(source, trialpython=["python2.3", "-Wall"], trial="./bin/trial", - # tests="foo.tests") does: - # python2.3 -Wall ./setup.py build - # PYTHONPATH=. python2.3 -Wall ./bin/trial -to foo.tests -</pre> - <p>If the output directory of <code>./setup.py build</code> is known, you can -pull the python code from the built location instead of the source -directories. This should be able to handle variations in where the -source comes from, as well as accomodating binary extension modules: - -<pre class="example"> # Trial(source,tests="toplevel.test",testpath='build/lib.linux-i686-2.3') - # does: - # python ./setup.py build - # PYTHONPATH=build/lib.linux-i686-2.3 trial -to toplevel.test -</pre> - <div class="node"> -<p><hr> -<a name="Status-Delivery"></a> -Next: <a rel="next" accesskey="n" href="#Command_002dline-tool">Command-line tool</a>, -Previous: <a rel="previous" accesskey="p" href="#Build-Process">Build Process</a>, -Up: <a rel="up" accesskey="u" href="#Top">Top</a> - -</div> - -<h2 class="chapter">7 Status Delivery</h2> - -<p>More details are available in the docstrings for each class, use a -command like <code>pydoc buildbot.status.html.WebStatus</code> to see them. -Most status delivery objects take a <code>categories=</code> argument, which -can contain a list of “category” names: in this case, it will only -show status for Builders that are in one of the named categories. - - <p>(implementor's note: each of these objects should be a -service.MultiService which will be attached to the BuildMaster object -when the configuration is processed. They should use -<code>self.parent.getStatus()</code> to get access to the top-level IStatus -object, either inside <code>startService</code> or later. They may call -<code>status.subscribe()</code> in <code>startService</code> to receive -notifications of builder events, in which case they must define -<code>builderAdded</code> and related methods. See the docstrings in -<samp><span class="file">buildbot/interfaces.py</span></samp> for full details.) - -<ul class="menu"> -<li><a accesskey="1" href="#WebStatus">WebStatus</a> -<li><a accesskey="2" href="#MailNotifier">MailNotifier</a> -<li><a accesskey="3" href="#IRC-Bot">IRC Bot</a> -<li><a accesskey="4" href="#PBListener">PBListener</a> -<li><a accesskey="5" href="#Writing-New-Status-Plugins">Writing New Status Plugins</a> -</ul> - -<!-- @node Email Delivery, , Status Delivery, Status Delivery --> -<!-- @subsection Email Delivery --> -<!-- DOCUMENT THIS --> -<div class="node"> -<p><hr> -<a name="WebStatus"></a> -Next: <a rel="next" accesskey="n" href="#MailNotifier">MailNotifier</a>, -Previous: <a rel="previous" accesskey="p" href="#Status-Delivery">Status Delivery</a>, -Up: <a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a> - -</div> - -<h3 class="section">7.1 WebStatus</h3> - -<p><a name="index-WebStatus-108"></a><a name="index-buildbot_002estatus_002eweb_002ebaseweb_002eWebStatus-109"></a> -The <code>buildbot.status.html.WebStatus</code> status target runs a small -web server inside the buildmaster. You can point a browser at this web -server and retrieve information about every build the buildbot knows -about, as well as find out what the buildbot is currently working on. - - <p>The first page you will see is the “Welcome Page”, which contains -links to all the other useful pages. This page is simply served from -the <samp><span class="file">public_html/index.html</span></samp> file in the buildmaster's base -directory, where it is created by the <samp><span class="command">buildbot create-master</span></samp> -command along with the rest of the buildmaster. - - <p>The most complex resource provided by <code>WebStatus</code> is the -“Waterfall Display”, which shows a time-based chart of events. This -somewhat-busy display provides detailed information about all steps of -all recent builds, and provides hyperlinks to look at individual build -logs and source changes. By simply reloading this page on a regular -basis, you will see a complete description of everything the buildbot -is currently working on. - - <p>There are also pages with more specialized information. For example, -there is a page which shows the last 20 builds performed by the -buildbot, one line each. Each line is a link to detailed information -about that build. By adding query arguments to the URL used to reach -this page, you can narrow the display to builds that involved certain -branches, or which ran on certain Builders. These pages are described -in great detail below. - - <p>When the buildmaster is created, a subdirectory named -<samp><span class="file">public_html/</span></samp> is created in its base directory. By default, <code>WebStatus</code> -will serve files from this directory: for example, when a user points -their browser at the buildbot's <code>WebStatus</code> URL, they will see -the contents of the <samp><span class="file">public_html/index.html</span></samp> file. Likewise, -<samp><span class="file">public_html/robots.txt</span></samp>, <samp><span class="file">public_html/buildbot.css</span></samp>, and -<samp><span class="file">public_html/favicon.ico</span></samp> are all useful things to have in there. -The first time a buildmaster is created, the <samp><span class="file">public_html</span></samp> -directory is populated with some sample files, which you will probably -want to customize for your own project. These files are all static: -the buildbot does not modify them in any way as it serves them to HTTP -clients. - -<pre class="example"> from buildbot.status.html import WebStatus - c['status'].append(WebStatus(8080)) -</pre> - <p>Note that the initial robots.txt file has Disallow lines for all of -the dynamically-generated buildbot pages, to discourage web spiders -and search engines from consuming a lot of CPU time as they crawl -through the entire history of your buildbot. If you are running the -buildbot behind a reverse proxy, you'll probably need to put the -robots.txt file somewhere else (at the top level of the parent web -server), and replace the URL prefixes in it with more suitable values. - - <p>If you would like to use an alternative root directory, add the -<code>public_html=..</code> option to the <code>WebStatus</code> creation: - -<pre class="example"> c['status'].append(WebStatus(8080, public_html="/var/www/buildbot")) -</pre> - <p>In addition, if you are familiar with twisted.web <em>Resource -Trees</em>, you can write code to add additional pages at places inside -this web space. Just use <code>webstatus.putChild</code> to place these -resources. - - <p>The following section describes the special URLs and the status views -they provide. - -<ul class="menu"> -<li><a accesskey="1" href="#WebStatus-Configuration-Parameters">WebStatus Configuration Parameters</a> -<li><a accesskey="2" href="#Buildbot-Web-Resources">Buildbot Web Resources</a> -<li><a accesskey="3" href="#XMLRPC-server">XMLRPC server</a> -<li><a accesskey="4" href="#HTML-Waterfall">HTML Waterfall</a> -</ul> - -<div class="node"> -<p><hr> -<a name="WebStatus-Configuration-Parameters"></a> -Next: <a rel="next" accesskey="n" href="#Buildbot-Web-Resources">Buildbot Web Resources</a>, -Previous: <a rel="previous" accesskey="p" href="#WebStatus">WebStatus</a>, -Up: <a rel="up" accesskey="u" href="#WebStatus">WebStatus</a> - -</div> - -<h4 class="subsection">7.1.1 WebStatus Configuration Parameters</h4> - -<p>The most common way to run a <code>WebStatus</code> is on a regular TCP -port. To do this, just pass in the TCP port number when you create the -<code>WebStatus</code> instance; this is called the <code>http_port</code> argument: - -<pre class="example"> from buildbot.status.html import WebStatus - c['status'].append(WebStatus(8080)) -</pre> - <p>The <code>http_port</code> argument is actually a “strports specification” -for the port that the web server should listen on. This can be a -simple port number, or a string like -<code>tcp:8080:interface=127.0.0.1</code> (to limit connections to the -loopback interface, and therefore to clients running on the same -host)<a rel="footnote" href="#fn-13" name="fnd-13"><sup>13</sup></a>. - - <p>If instead (or in addition) you provide the <code>distrib_port</code> -argument, a twisted.web distributed server will be started either on a -TCP port (if <code>distrib_port</code> is like <code>"tcp:12345"</code>) or more -likely on a UNIX socket (if <code>distrib_port</code> is like -<code>"unix:/path/to/socket"</code>). - - <p>The <code>distrib_port</code> option means that, on a host with a -suitably-configured twisted-web server, you do not need to consume a -separate TCP port for the buildmaster's status web page. When the web -server is constructed with <code>mktap web --user</code>, URLs that point to -<code>http://host/~username/</code> are dispatched to a sub-server that is -listening on a UNIX socket at <code>~username/.twisted-web-pb</code>. On -such a system, it is convenient to create a dedicated <code>buildbot</code> -user, then set <code>distrib_port</code> to -<code>"unix:"+os.path.expanduser("~/.twistd-web-pb")</code>. This -configuration will make the HTML status page available at -<code>http://host/~buildbot/</code> . Suitable URL remapping can make it -appear at <code>http://host/buildbot/</code>, and the right virtual host -setup can even place it at <code>http://buildbot.host/</code> . - - <p>The other <code>WebStatus</code> argument is <code>allowForce</code>. If set to -True, then the web page will provide a “Force Build” button that -allows visitors to manually trigger builds. This is useful for -developers to re-run builds that have failed because of intermittent -problems in the test suite, or because of libraries that were not -installed at the time of the previous build. You may not wish to allow -strangers to cause a build to run: in that case, set this to False to -remove these buttons. The default value is False. - -<div class="node"> -<p><hr> -<a name="Buildbot-Web-Resources"></a> -Next: <a rel="next" accesskey="n" href="#XMLRPC-server">XMLRPC server</a>, -Previous: <a rel="previous" accesskey="p" href="#WebStatus-Configuration-Parameters">WebStatus Configuration Parameters</a>, -Up: <a rel="up" accesskey="u" href="#WebStatus">WebStatus</a> - -</div> - -<h4 class="subsection">7.1.2 Buildbot Web Resources</h4> - -<p>Certain URLs are “magic”, and the pages they serve are created by -code in various classes in the <samp><span class="file">buildbot.status.web</span></samp> package -instead of being read from disk. The most common way to access these -pages is for the buildmaster admin to write or modify the -<samp><span class="file">index.html</span></samp> page to contain links to them. Of course other -project web pages can contain links to these buildbot pages as well. - - <p>Many pages can be modified by adding query arguments to the URL. For -example, a page which shows the results of the most recent build -normally does this for all builders at once. But by appending -“?builder=i386” to the end of the URL, the page will show only the -results for the “i386” builder. When used in this way, you can add -multiple “builder=” arguments to see multiple builders. Remembering -that URL query arguments are separated <em>from each other</em> with -ampersands, a URL that ends in “?builder=i386&builder=ppc” would -show builds for just those two Builders. - - <p>The <code>branch=</code> query argument can be used on some pages. This -filters the information displayed by that page down to only the builds -or changes which involved the given branch. Use <code>branch=trunk</code> to -reference the trunk: if you aren't intentionally using branches, -you're probably using trunk. Multiple <code>branch=</code> arguments can be -used to examine multiple branches at once (so appending -<code>?branch=foo&branch=bar</code> to the URL will show builds involving -either branch). No <code>branch=</code> arguments means to show builds and -changes for all branches. - - <p>Some pages may include the Builder name or the build number in the -main part of the URL itself. For example, a page that describes Build -#7 of the “i386” builder would live at <samp><span class="file">/builders/i386/builds/7</span></samp>. - - <p>The table below lists all of the internal pages and the URLs that can -be used to access them. - - <p>NOTE: of the pages described here, <code>/slave_status_timeline</code> and -<code>/last_build</code> have not yet been implemented, and <code>/xmlrpc</code> -has only a few methods so far. Future releases will improve this. - - <dl> -<dt><code>/waterfall</code><dd> -This provides a chronologically-oriented display of the activity of -all builders. It is the same display used by the Waterfall display. - - <p>By adding one or more “builder=” query arguments, the Waterfall is -restricted to only showing information about the given Builders. By -adding one or more “branch=” query arguments, the display is -restricted to showing information about the given branches. In -addition, adding one or more “category=” query arguments to the URL -will limit the display to Builders that were defined with one of the -given categories. - - <p>A 'show_events=true' query argument causes the display to include -non-Build events, like slaves attaching and detaching, as well as -reconfiguration events. 'show_events=false' hides these events. The -default is to show them. - - <p>The <code>last_time=</code>, <code>first_time=</code>, and <code>show_time=</code> -arguments will control what interval of time is displayed. The default -is to show the latest events, but these can be used to look at earlier -periods in history. The <code>num_events=</code> argument also provides a -limit on the size of the displayed page. - - <p>The Waterfall has references to resources many of the other portions -of the URL space: <samp><span class="file">/builders</span></samp> for access to individual builds, -<samp><span class="file">/changes</span></samp> for access to information about source code changes, -etc. - - <br><dt><code>/rss</code><dd> -This provides a rss feed summarizing all failed builds. The same -query-arguments used by 'waterfall' can be added to filter the -feed output. - - <br><dt><code>/atom</code><dd> -This provides an atom feed summarizing all failed builds. The same -query-arguments used by 'waterfall' can be added to filter the feed -output. - - <br><dt><code>/builders/$BUILDERNAME</code><dd> -This describes the given Builder, and provides buttons to force a build. - - <br><dt><code>/builders/$BUILDERNAME/builds/$BUILDNUM</code><dd> -This describes a specific Build. - - <br><dt><code>/builders/$BUILDERNAME/builds/$BUILDNUM/steps/$STEPNAME</code><dd> -This describes a specific BuildStep. - - <br><dt><code>/builders/$BUILDERNAME/builds/$BUILDNUM/steps/$STEPNAME/logs/$LOGNAME</code><dd> -This provides an HTML representation of a specific logfile. - - <br><dt><code>/builders/$BUILDERNAME/builds/$BUILDNUM/steps/$STEPNAME/logs/$LOGNAME/text</code><dd> -This returns the logfile as plain text, without any HTML coloring -markup. It also removes the “headers”, which are the lines that -describe what command was run and what the environment variable -settings were like. This maybe be useful for saving to disk and -feeding to tools like 'grep'. - - <br><dt><code>/changes</code><dd> -This provides a brief description of the ChangeSource in use -(see <a href="#Change-Sources">Change Sources</a>). - - <br><dt><code>/changes/NN</code><dd> -This shows detailed information about the numbered Change: who was the -author, what files were changed, what revision number was represented, -etc. - - <br><dt><code>/buildslaves</code><dd> -This summarizes each BuildSlave, including which Builders are -configured to use it, whether the buildslave is currently connected or -not, and host information retrieved from the buildslave itself. - - <br><dt><code>/one_line_per_build</code><dd> -This page shows one line of text for each build, merging information -from all Builders<a rel="footnote" href="#fn-14" name="fnd-14"><sup>14</sup></a>. Each line specifies -the name of the Builder, the number of the Build, what revision it -used, and a summary of the results. Successful builds are in green, -while failing builds are in red. The date and time of the build are -added to the right-hand edge of the line. The lines are ordered by -build finish timestamp. - - <p>One or more <code>builder=</code> or <code>branch=</code> arguments can be used to -restrict the list. In addition, a <code>numbuilds=</code> argument will -control how many lines are displayed (20 by default). - - <br><dt><code>/one_box_per_builder</code><dd> -This page shows a small table, with one box for each Builder, -containing the results of the most recent Build. It does not show the -individual steps, or the current status. This is a simple summary of -buildbot status: if this page is green, then all tests are passing. - - <p>As with <code>/one_line_per_build</code>, this page will also honor -<code>builder=</code> and <code>branch=</code> arguments. - - <br><dt><code>/about</code><dd> -This page gives a brief summary of the Buildbot itself: software -version, versions of some libraries that the Buildbot depends upon, -etc. It also contains a link to the buildbot.net home page. - - <br><dt><code>/slave_status_timeline</code><dd> -(note: this page has not yet been implemented) - - <p>This provides a chronological display of configuration and operational -events: master startup/shutdown, slave connect/disconnect, and -config-file changes. When a config-file reload is abandoned because of -an error in the config file, the error is displayed on this page. - - <p>This page does not show any builds. - - <br><dt><code>/last_build/$BUILDERNAME/status.png</code><dd> -This returns a PNG image that describes the results of the most recent -build, which can be referenced in an IMG tag by other pages, perhaps -from a completely different site. Use it as you would a webcounter. - - </dl> - - <p>There are also a set of web-status resources that are intended for use -by other programs, rather than humans. - - <dl> -<dt><code>/xmlrpc</code><dd> -This runs an XML-RPC server which can be used to query status -information about various builds. See <a href="#XMLRPC-server">XMLRPC server</a> for more -details. - - </dl> - -<div class="node"> -<p><hr> -<a name="XMLRPC-server"></a> -Next: <a rel="next" accesskey="n" href="#HTML-Waterfall">HTML Waterfall</a>, -Previous: <a rel="previous" accesskey="p" href="#Buildbot-Web-Resources">Buildbot Web Resources</a>, -Up: <a rel="up" accesskey="u" href="#WebStatus">WebStatus</a> - -</div> - -<h4 class="subsection">7.1.3 XMLRPC server</h4> - -<p>When using WebStatus, the buildbot runs an XML-RPC server at -<samp><span class="file">/xmlrpc</span></samp> that can be used by other programs to query build -status. The following table lists the methods that can be invoked -using this interface. - - <dl> -<dt><code>getAllBuildsInInterval(start, stop)</code><dd> -Return a list of builds that have completed after the 'start' -timestamp and before the 'stop' timestamp. This looks at all Builders. - - <p>The timestamps are integers, interpreted as standard unix timestamps -(seconds since epoch). - - <p>Each Build is returned as a tuple in the form: <code>(buildername, -buildnumber, build_end, branchname, revision, results, text)</code> - - <p>The buildnumber is an integer. 'build_end' is an integer (seconds -since epoch) specifying when the build finished. - - <p>The branchname is a string, which may be an empty string to indicate -None (i.e. the default branch). The revision is a string whose meaning -is specific to the VC system in use, and comes from the 'got_revision' -build property. The results are expressed as a string, one of -('success', 'warnings', 'failure', 'exception'). The text is a list of -short strings that ought to be joined by spaces and include slightly -more data about the results of the build. - - <br><dt><code>getBuild(builder_name, build_number)</code><dd> -Return information about a specific build. - - <p>This returns a dictionary (aka “struct” in XMLRPC terms) with -complete information about the build. It does not include the contents -of the log files, but it has just about everything else. - - </dl> - -<div class="node"> -<p><hr> -<a name="HTML-Waterfall"></a> -Previous: <a rel="previous" accesskey="p" href="#XMLRPC-server">XMLRPC server</a>, -Up: <a rel="up" accesskey="u" href="#WebStatus">WebStatus</a> - -</div> - -<h4 class="subsection">7.1.4 HTML Waterfall</h4> - -<p><a name="index-Waterfall-110"></a><a name="index-buildbot_002estatus_002ehtml_002eWaterfall-111"></a> -The <code>Waterfall</code> status target, deprecated as of 0.7.6, is a -subset of the regular <code>WebStatus</code> resource (see <a href="#WebStatus">WebStatus</a>). -This section (and the <code>Waterfall</code> class itself) will be removed -from a future release. - -<pre class="example"> from buildbot.status import html - w = html.WebStatus(http_port=8080) - c['status'].append(w) -</pre> - <div class="node"> -<p><hr> -<a name="MailNotifier"></a> -Next: <a rel="next" accesskey="n" href="#IRC-Bot">IRC Bot</a>, -Previous: <a rel="previous" accesskey="p" href="#WebStatus">WebStatus</a>, -Up: <a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a> - -</div> - -<h3 class="section">7.2 MailNotifier</h3> - -<p><a name="index-email-112"></a><a name="index-mail-113"></a><a name="index-buildbot_002estatus_002email_002eMailNotifier-114"></a> -The buildbot can also send email when builds finish. The most common -use of this is to tell developers when their change has caused the -build to fail. It is also quite common to send a message to a mailing -list (usually named “builds” or similar) about every build. - - <p>The <code>MailNotifier</code> status target is used to accomplish this. You -configure it by specifying who mail should be sent to, under what -circumstances mail should be sent, and how to deliver the mail. It can -be configured to only send out mail for certain builders, and only -send messages when the build fails, or when the builder transitions -from success to failure. It can also be configured to include various -build logs in each message. - - <p>By default, the message will be sent to the Interested Users list -(see <a href="#Doing-Things-With-Users">Doing Things With Users</a>), which includes all developers who -made changes in the build. You can add additional recipients with the -extraRecipients argument. - - <p>Each MailNotifier sends mail to a single set of recipients. To send -different kinds of mail to different recipients, use multiple -MailNotifiers. - - <p>The following simple example will send an email upon the completion of -each build, to just those developers whose Changes were included in -the build. The email contains a description of the Build, its results, -and URLs where more information can be obtained. - -<pre class="example"> from buildbot.status.mail import MailNotifier - mn = MailNotifier(fromaddr="buildbot@example.org", lookup="example.org") - c['status'].append(mn) -</pre> - <p>To get a simple one-message-per-build (say, for a mailing list), use -the following form instead. This form does not send mail to individual -developers (and thus does not need the <code>lookup=</code> argument, -explained below), instead it only ever sends mail to the “extra -recipients” named in the arguments: - -<pre class="example"> mn = MailNotifier(fromaddr="buildbot@example.org", - sendToInterestedUsers=False, - extraRecipients=['listaddr@example.org']) -</pre> - <p>In some cases it is desirable to have different information then what -is provided in a standard MailNotifier message. For this purpose -MailNotifier provides the argument customMesg (a function) which allows -for the creation of messages with unique content. - - <p>For example it can be useful to display the last few lines of a log file -and recent changes when a builder fails: - -<pre class="example"> def message(attrs): - logLines = 10 - text = list() - text.append("STATUS: %s" % attrs['result'].title()) - text.append("") - text.extend([c.asText() for c in attrs['changes']]) - text.append("") - name, url, lines = attrs['logs'][-1] - text.append("Last %d lines of '%s':" % (logLines, name)) - text.extend(["\t%s\n" % line for line in lines[len(lines)-logLines:]]) - text.append("") - text.append("-buildbot") - return ("\n".join(text), 'plain') - - mn = MailNotifier(fromaddr="buildbot@example.org", - sendToInterestedUsers=False, - mode='problem', - extraRecipients=['listaddr@example.org'], - customMesg=message) -</pre> - <p>A customMesg function takes a single dict argument (see below) and returns a -tuple of strings. The first string is the complete text of the message and the -second is the message type ('plain' or 'html'). The 'html' type should be used -when generating an HTML message: - -<pre class="example"> def message(attrs): - logLines = 10 - text = list() - text.append('<h4>Build status %s.</h4>' % (attrs['result'].title())) - if attrs['changes']: - text.append('<h4>Recent Changes:</h4>') - text.extend([c.asHTML() for c in attrs['changes']]) - name, url, lines = attrs['logs'][-1] - text.append('<h4>Last %d lines of "%s":</h4>' % (logLines, name)) - text.append('<p>') - text.append('<br>'.join([line for line in lines[len(lines)-logLines:]])) - text.append('</p>') - text.append('<br><br>') - text.append('Full log at: %s' % url) - text.append('<br><br>') - text.append('<b>-buildbot</b>') - return ('\n'.join(text), 'html') -</pre> - <h3 class="heading">MailNotifier arguments</h3> - - <dl> -<dt><code>fromaddr</code><dd>The email address to be used in the 'From' header. - - <br><dt><code>sendToInterestedUsers</code><dd>(boolean). If True (the default), send mail to all of the Interested -Users. If False, only send mail to the extraRecipients list. - - <br><dt><code>extraRecipients</code><dd>(tuple of strings). A list of email addresses to which messages should -be sent (in addition to the InterestedUsers list, which includes any -developers who made Changes that went into this build). It is a good -idea to create a small mailing list and deliver to that, then let -subscribers come and go as they please. - - <br><dt><code>subject</code><dd>(string). A string to be used as the subject line of the message. -<code>%(builder)s</code> will be replaced with the name of the builder which -provoked the message. - - <br><dt><code>mode</code><dd>(string). Default to 'all'. One of: - <dl> -<dt><code>all</code><dd>Send mail about all builds, bothpassing and failing -<br><dt><code>failing</code><dd>Only send mail about builds which fail -<br><dt><code>problem</code><dd>Only send mail about a build which failed when the previous build has passed. -If your builds usually pass, then this will only send mail when a problem -occurs. -</dl> - - <br><dt><code>builders</code><dd>(list of strings). A list of builder names for which mail should be -sent. Defaults to None (send mail for all builds). Use either builders -or categories, but not both. - - <br><dt><code>categories</code><dd>(list of strings). A list of category names to serve status -information for. Defaults to None (all categories). Use either -builders or categories, but not both. - - <br><dt><code>addLogs</code><dd>(boolean). If True, include all build logs as attachments to the -messages. These can be quite large. This can also be set to a list of -log names, to send a subset of the logs. Defaults to False. - - <br><dt><code>relayhost</code><dd>(string). The host to which the outbound SMTP connection should be -made. Defaults to 'localhost' - - <br><dt><code>lookup</code><dd>(implementor of <code>IEmailLookup</code>). Object which provides -IEmailLookup, which is responsible for mapping User names (which come -from the VC system) into valid email addresses. If not provided, the -notifier will only be able to send mail to the addresses in the -extraRecipients list. Most of the time you can use a simple Domain -instance. As a shortcut, you can pass as string: this will be treated -as if you had provided Domain(str). For example, -lookup='twistedmatrix.com' will allow mail to be sent to all -developers whose SVN usernames match their twistedmatrix.com account -names. See buildbot/status/mail.py for more details. - - <br><dt><code>customMesg</code><dd>This is a optional function that can be used to generate a custom mail -message. The customMesg function takes a single dict and must return a -tuple containing the message text and type ('html' or 'plain'). Below is a list -of availale keys in the dict passed to customMesg: - - <dl> -<dt><code>builderName</code><dd>(str) Name of the builder that generated this event. -<br><dt><code>projectName</code><dd>(str) Name of the project. -<br><dt><code>mode</code><dd>(str) Mode set in MailNotifier. (failing, passing, problem). -<br><dt><code>result</code><dd>(str) Builder result as a string. 'success', 'warnings', 'failure', 'skipped', or 'exception' -<br><dt><code>buildURL</code><dd>(str) URL to build page. -<br><dt><code>buildbotURL</code><dd>(str) URL to buildbot main page. -<br><dt><code>buildText</code><dd>(str) Build text from build.getText(). -<br><dt><code>slavename</code><dd>(str) Slavename. -<br><dt><code>reason</code><dd>(str) Build reason from build.getReason(). -<br><dt><code>responsibleUsers</code><dd>(List of str) List of responsible users. -<br><dt><code>branch</code><dd>(str) Name of branch used. If no SourceStamp exists branch -is an empty string. -<br><dt><code>revision</code><dd>(str) Name of revision used. If no SourceStamp exists revision -is an empty string. -<br><dt><code>patch</code><dd>(str) Name of patch used. If no SourceStamp exists patch -is an empty string. -<br><dt><code>changes</code><dd>(list of objs) List of change objects from SourceStamp. A change -object has the following useful information: - <dl> -<dt><code>who</code><dd>(str) who made this change -<br><dt><code>revision</code><dd>(str) what VC revision is this change -<br><dt><code>branch</code><dd>(str) on what branch did this change occur -<br><dt><code>when</code><dd>(str) when did this change occur -<br><dt><code>files</code><dd>(list of str) what files were affected in this change -<br><dt><code>comments</code><dd>(str) comments reguarding the change. -</dl> - The functions asText and asHTML return a list of strings with -the above information formatted. -<br><dt><code>logs</code><dd>(List of Tuples) List of tuples where each tuple contains the log name, log url, -and log contents as a list of strings. -</dl> - </dl> - -<div class="node"> -<p><hr> -<a name="IRC-Bot"></a> -Next: <a rel="next" accesskey="n" href="#PBListener">PBListener</a>, -Previous: <a rel="previous" accesskey="p" href="#MailNotifier">MailNotifier</a>, -Up: <a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a> - -</div> - -<h3 class="section">7.3 IRC Bot</h3> - -<p><a name="index-IRC-115"></a><a name="index-buildbot_002estatus_002ewords_002eIRC-116"></a> - - <p>The <code>buildbot.status.words.IRC</code> status target creates an IRC bot -which will attach to certain channels and be available for status -queries. It can also be asked to announce builds as they occur, or be -told to shut up. - -<pre class="example"> from buildbot.status import words - irc = words.IRC("irc.example.org", "botnickname", - channels=["channel1", "channel2"], - password="mysecretpassword", - notify_events={ - 'exception': 1, - 'successToFailure': 1, - 'failureToSuccess': 1, - }) - c['status'].append(irc) -</pre> - <p>Take a look at the docstring for <code>words.IRC</code> for more details on -configuring this service. The <code>password</code> argument, if provided, -will be sent to Nickserv to claim the nickname: some IRC servers will -not allow clients to send private messages until they have logged in -with a password. - - <p>To use the service, you address messages at the buildbot, either -normally (<code>botnickname: status</code>) or with private messages -(<code>/msg botnickname status</code>). The buildbot will respond in kind. - - <p>Some of the commands currently available: - - <dl> -<dt><code>list builders</code><dd>Emit a list of all configured builders -<br><dt><code>status BUILDER</code><dd>Announce the status of a specific Builder: what it is doing right now. -<br><dt><code>status all</code><dd>Announce the status of all Builders -<br><dt><code>watch BUILDER</code><dd>If the given Builder is currently running, wait until the Build is -finished and then announce the results. -<br><dt><code>last BUILDER</code><dd>Return the results of the last build to run on the given Builder. -<br><dt><code>join CHANNEL</code><dd>Join the given IRC channel -<br><dt><code>leave CHANNEL</code><dd>Leave the given IRC channel -<br><dt><code>notify on|off|list EVENT</code><dd>Report events relating to builds. If the command is issued as a -private message, then the report will be sent back as a private -message to the user who issued the command. Otherwise, the report -will be sent to the channel. Available events to be notified are: - - <dl> -<dt><code>started</code><dd>A build has started -<br><dt><code>finished</code><dd>A build has finished -<br><dt><code>success</code><dd>A build finished successfully -<br><dt><code>failed</code><dd>A build failed -<br><dt><code>exception</code><dd>A build generated and exception -<br><dt><code>xToY</code><dd>The previous build was x, but this one is Y, where x and Y are each -one of success, warnings, failure, exception (except Y is -capitalized). For example: successToFailure will notify if the -previous build was successful, but this one failed -</dl> - - <br><dt><code>help COMMAND</code><dd>Describe a command. Use <code>help commands</code> to get a list of known -commands. -<br><dt><code>source</code><dd>Announce the URL of the Buildbot's home page. -<br><dt><code>version</code><dd>Announce the version of this Buildbot. -</dl> - - <p>Additionally, the config file may specify default notification options -as shown in the example earlier. - - <p>If the <code>allowForce=True</code> option was used, some addtional commands -will be available: - - <dl> -<dt><code>force build BUILDER REASON</code><dd>Tell the given Builder to start a build of the latest code. The user -requesting the build and REASON are recorded in the Build status. The -buildbot will announce the build's status when it finishes. - - <br><dt><code>stop build BUILDER REASON</code><dd>Terminate any running build in the given Builder. REASON will be added -to the build status to explain why it was stopped. You might use this -if you committed a bug, corrected it right away, and don't want to -wait for the first build (which is destined to fail) to complete -before starting the second (hopefully fixed) build. -</dl> - -<div class="node"> -<p><hr> -<a name="PBListener"></a> -Next: <a rel="next" accesskey="n" href="#Writing-New-Status-Plugins">Writing New Status Plugins</a>, -Previous: <a rel="previous" accesskey="p" href="#IRC-Bot">IRC Bot</a>, -Up: <a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a> - -</div> - -<h3 class="section">7.4 PBListener</h3> - -<p><a name="index-PBListener-117"></a><a name="index-buildbot_002estatus_002eclient_002ePBListener-118"></a> - -<pre class="example"> import buildbot.status.client - pbl = buildbot.status.client.PBListener(port=int, user=str, - passwd=str) - c['status'].append(pbl) -</pre> - <p>This sets up a PB listener on the given TCP port, to which a PB-based -status client can connect and retrieve status information. -<code>buildbot statusgui</code> (see <a href="#statusgui">statusgui</a>) is an example of such a -status client. The <code>port</code> argument can also be a strports -specification string. - -<div class="node"> -<p><hr> -<a name="Writing-New-Status-Plugins"></a> -Previous: <a rel="previous" accesskey="p" href="#PBListener">PBListener</a>, -Up: <a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a> - -</div> - -<h3 class="section">7.5 Writing New Status Plugins</h3> - -<p>TODO: this needs a lot more examples - - <p>Each status plugin is an object which provides the -<code>twisted.application.service.IService</code> interface, which creates a -tree of Services with the buildmaster at the top [not strictly true]. -The status plugins are all children of an object which implements -<code>buildbot.interfaces.IStatus</code>, the main status object. From this -object, the plugin can retrieve anything it wants about current and -past builds. It can also subscribe to hear about new and upcoming -builds. - - <p>Status plugins which only react to human queries (like the Waterfall -display) never need to subscribe to anything: they are idle until -someone asks a question, then wake up and extract the information they -need to answer it, then they go back to sleep. Plugins which need to -act spontaneously when builds complete (like the MailNotifier plugin) -need to subscribe to hear about new builds. - - <p>If the status plugin needs to run network services (like the HTTP -server used by the Waterfall plugin), they can be attached as Service -children of the plugin itself, using the <code>IServiceCollection</code> -interface. - -<div class="node"> -<p><hr> -<a name="Command-line-tool"></a> -<a name="Command_002dline-tool"></a> -Next: <a rel="next" accesskey="n" href="#Resources">Resources</a>, -Previous: <a rel="previous" accesskey="p" href="#Status-Delivery">Status Delivery</a>, -Up: <a rel="up" accesskey="u" href="#Top">Top</a> - -</div> - -<h2 class="chapter">8 Command-line tool</h2> - -<p>The <samp><span class="command">buildbot</span></samp> command-line tool can be used to start or stop a -buildmaster or buildbot, and to interact with a running buildmaster. -Some of its subcommands are intended for buildmaster admins, while -some are for developers who are editing the code that the buildbot is -monitoring. - -<ul class="menu"> -<li><a accesskey="1" href="#Administrator-Tools">Administrator Tools</a> -<li><a accesskey="2" href="#Developer-Tools">Developer Tools</a> -<li><a accesskey="3" href="#Other-Tools">Other Tools</a> -<li><a accesskey="4" href="#g_t_002ebuildbot-config-directory">.buildbot config directory</a> -</ul> - -<div class="node"> -<p><hr> -<a name="Administrator-Tools"></a> -Next: <a rel="next" accesskey="n" href="#Developer-Tools">Developer Tools</a>, -Previous: <a rel="previous" accesskey="p" href="#Command_002dline-tool">Command-line tool</a>, -Up: <a rel="up" accesskey="u" href="#Command_002dline-tool">Command-line tool</a> - -</div> - -<h3 class="section">8.1 Administrator Tools</h3> - -<p>The following <samp><span class="command">buildbot</span></samp> sub-commands are intended for -buildmaster administrators: - -<h3 class="heading">create-master</h3> - -<p>This creates a new directory and populates it with files that allow it -to be used as a buildmaster's base directory. - -<pre class="example"> buildbot create-master BASEDIR -</pre> - <h3 class="heading">create-slave</h3> - -<p>This creates a new directory and populates it with files that let it -be used as a buildslave's base directory. You must provide several -arguments, which are used to create the initial <samp><span class="file">buildbot.tac</span></samp> -file. - -<pre class="example"> buildbot create-slave <var>BASEDIR</var> <var>MASTERHOST</var>:<var>PORT</var> <var>SLAVENAME</var> <var>PASSWORD</var> -</pre> - <h3 class="heading">start</h3> - -<p>This starts a buildmaster or buildslave which was already created in -the given base directory. The daemon is launched in the background, -with events logged to a file named <samp><span class="file">twistd.log</span></samp>. - -<pre class="example"> buildbot start BASEDIR -</pre> - <h3 class="heading">stop</h3> - -<p>This terminates the daemon (either buildmaster or buildslave) running -in the given directory. - -<pre class="example"> buildbot stop BASEDIR -</pre> - <h3 class="heading">sighup</h3> - -<p>This sends a SIGHUP to the buildmaster running in the given directory, -which causes it to re-read its <samp><span class="file">master.cfg</span></samp> file. - -<pre class="example"> buildbot sighup BASEDIR -</pre> - <div class="node"> -<p><hr> -<a name="Developer-Tools"></a> -Next: <a rel="next" accesskey="n" href="#Other-Tools">Other Tools</a>, -Previous: <a rel="previous" accesskey="p" href="#Administrator-Tools">Administrator Tools</a>, -Up: <a rel="up" accesskey="u" href="#Command_002dline-tool">Command-line tool</a> - -</div> - -<h3 class="section">8.2 Developer Tools</h3> - -<p>These tools are provided for use by the developers who are working on -the code that the buildbot is monitoring. - -<ul class="menu"> -<li><a accesskey="1" href="#statuslog">statuslog</a> -<li><a accesskey="2" href="#statusgui">statusgui</a> -<li><a accesskey="3" href="#try">try</a> -</ul> - -<div class="node"> -<p><hr> -<a name="statuslog"></a> -Next: <a rel="next" accesskey="n" href="#statusgui">statusgui</a>, -Previous: <a rel="previous" accesskey="p" href="#Developer-Tools">Developer Tools</a>, -Up: <a rel="up" accesskey="u" href="#Developer-Tools">Developer Tools</a> - -</div> - -<h4 class="subsection">8.2.1 statuslog</h4> - -<pre class="example"> buildbot statuslog --master <var>MASTERHOST</var>:<var>PORT</var> -</pre> - <p>This command starts a simple text-based status client, one which just -prints out a new line each time an event occurs on the buildmaster. - - <p>The <samp><span class="option">--master</span></samp> option provides the location of the -<code>buildbot.status.client.PBListener</code> status port, used to deliver -build information to realtime status clients. The option is always in -the form of a string, with hostname and port number separated by a -colon (<code>HOSTNAME:PORTNUM</code>). Note that this port is <em>not</em> the -same as the slaveport (although a future version may allow the same -port number to be used for both purposes). If you get an error message -to the effect of “Failure: twisted.cred.error.UnauthorizedLogin:”, -this may indicate that you are connecting to the slaveport rather than -a <code>PBListener</code> port. - - <p>The <samp><span class="option">--master</span></samp> option can also be provided by the -<code>masterstatus</code> name in <samp><span class="file">.buildbot/options</span></samp> (see <a href="#g_t_002ebuildbot-config-directory">.buildbot config directory</a>). - -<div class="node"> -<p><hr> -<a name="statusgui"></a> -Next: <a rel="next" accesskey="n" href="#try">try</a>, -Previous: <a rel="previous" accesskey="p" href="#statuslog">statuslog</a>, -Up: <a rel="up" accesskey="u" href="#Developer-Tools">Developer Tools</a> - -</div> - -<h4 class="subsection">8.2.2 statusgui</h4> - -<p><a name="index-statusgui-119"></a> -If you have set up a PBListener (see <a href="#PBListener">PBListener</a>), you will be able -to monitor your Buildbot using a simple Gtk+ application invoked with -the <code>buildbot statusgui</code> command: - -<pre class="example"> buildbot statusgui --master <var>MASTERHOST</var>:<var>PORT</var> -</pre> - <p>This command starts a simple Gtk+-based status client, which contains -a few boxes for each Builder that change color as events occur. It -uses the same <samp><span class="option">--master</span></samp> argument as the <samp><span class="command">buildbot -statuslog</span></samp> command (see <a href="#statuslog">statuslog</a>). - -<div class="node"> -<p><hr> -<a name="try"></a> -Previous: <a rel="previous" accesskey="p" href="#statusgui">statusgui</a>, -Up: <a rel="up" accesskey="u" href="#Developer-Tools">Developer Tools</a> - -</div> - -<h4 class="subsection">8.2.3 try</h4> - -<p>This lets a developer to ask the question “What would happen if I -committed this patch right now?”. It runs the unit test suite (across -multiple build platforms) on the developer's current code, allowing -them to make sure they will not break the tree when they finally -commit their changes. - - <p>The <samp><span class="command">buildbot try</span></samp> command is meant to be run from within a -developer's local tree, and starts by figuring out the base revision -of that tree (what revision was current the last time the tree was -updated), and a patch that can be applied to that revision of the tree -to make it match the developer's copy. This (revision, patch) pair is -then sent to the buildmaster, which runs a build with that -SourceStamp. If you want, the tool will emit status messages as the -builds run, and will not terminate until the first failure has been -detected (or the last success). - - <p>There is an alternate form which accepts a pre-made patch file -(typically the output of a command like 'svn diff'). This “–diff” -form does not require a local tree to run from. See See <a href="#try-_002d_002ddiff">try –diff</a>. - - <p>For this command to work, several pieces must be in place: - -<h3 class="heading">TryScheduler</h3> - -<p><a name="index-buildbot_002escheduler_002eTry_005fJobdir-120"></a><a name="index-buildbot_002escheduler_002eTry_005fUserpass-121"></a> -The buildmaster must have a <code>scheduler.Try</code> instance in -the config file's <code>c['schedulers']</code> list. This lets the -administrator control who may initiate these “trial” builds, which -branches are eligible for trial builds, and which Builders should be -used for them. - - <p>The <code>TryScheduler</code> has various means to accept build requests: -all of them enforce more security than the usual buildmaster ports do. -Any source code being built can be used to compromise the buildslave -accounts, but in general that code must be checked out from the VC -repository first, so only people with commit privileges can get -control of the buildslaves. The usual force-build control channels can -waste buildslave time but do not allow arbitrary commands to be -executed by people who don't have those commit privileges. However, -the source code patch that is provided with the trial build does not -have to go through the VC system first, so it is important to make -sure these builds cannot be abused by a non-committer to acquire as -much control over the buildslaves as a committer has. Ideally, only -developers who have commit access to the VC repository would be able -to start trial builds, but unfortunately the buildmaster does not, in -general, have access to VC system's user list. - - <p>As a result, the <code>TryScheduler</code> requires a bit more -configuration. There are currently two ways to set this up: - - <dl> -<dt><strong>jobdir (ssh)</strong><dd> -This approach creates a command queue directory, called the -“jobdir”, in the buildmaster's working directory. The buildmaster -admin sets the ownership and permissions of this directory to only -grant write access to the desired set of developers, all of whom must -have accounts on the machine. The <code>buildbot try</code> command creates -a special file containing the source stamp information and drops it in -the jobdir, just like a standard maildir. When the buildmaster notices -the new file, it unpacks the information inside and starts the builds. - - <p>The config file entries used by 'buildbot try' either specify a local -queuedir (for which write and mv are used) or a remote one (using scp -and ssh). - - <p>The advantage of this scheme is that it is quite secure, the -disadvantage is that it requires fiddling outside the buildmaster -config (to set the permissions on the jobdir correctly). If the -buildmaster machine happens to also house the VC repository, then it -can be fairly easy to keep the VC userlist in sync with the -trial-build userlist. If they are on different machines, this will be -much more of a hassle. It may also involve granting developer accounts -on a machine that would not otherwise require them. - - <p>To implement this, the buildslave invokes 'ssh -l username host -buildbot tryserver ARGS', passing the patch contents over stdin. The -arguments must include the inlet directory and the revision -information. - - <br><dt><strong>user+password (PB)</strong><dd> -In this approach, each developer gets a username/password pair, which -are all listed in the buildmaster's configuration file. When the -developer runs <code>buildbot try</code>, their machine connects to the -buildmaster via PB and authenticates themselves using that username -and password, then sends a PB command to start the trial build. - - <p>The advantage of this scheme is that the entire configuration is -performed inside the buildmaster's config file. The disadvantages are -that it is less secure (while the “cred” authentication system does -not expose the password in plaintext over the wire, it does not offer -most of the other security properties that SSH does). In addition, the -buildmaster admin is responsible for maintaining the username/password -list, adding and deleting entries as developers come and go. - - </dl> - - <p>For example, to set up the “jobdir” style of trial build, using a -command queue directory of <samp><span class="file">MASTERDIR/jobdir</span></samp> (and assuming that -all your project developers were members of the <code>developers</code> unix -group), you would first create that directory (with <samp><span class="command">mkdir -MASTERDIR/jobdir MASTERDIR/jobdir/new MASTERDIR/jobdir/cur -MASTERDIR/jobdir/tmp; chgrp developers MASTERDIR/jobdir -MASTERDIR/jobdir/*; chmod g+rwx,o-rwx MASTERDIR/jobdir -MASTERDIR/jobdir/*</span></samp>), and then use the following scheduler in the -buildmaster's config file: - -<pre class="example"> from buildbot.scheduler import Try_Jobdir - s = Try_Jobdir("try1", ["full-linux", "full-netbsd", "full-OSX"], - jobdir="jobdir") - c['schedulers'] = [s] -</pre> - <p>Note that you must create the jobdir before telling the buildmaster to -use this configuration, otherwise you will get an error. Also remember -that the buildmaster must be able to read and write to the jobdir as -well. Be sure to watch the <samp><span class="file">twistd.log</span></samp> file (see <a href="#Logfiles">Logfiles</a>) -as you start using the jobdir, to make sure the buildmaster is happy -with it. - - <p>To use the username/password form of authentication, create a -<code>Try_Userpass</code> instance instead. It takes the same -<code>builderNames</code> argument as the <code>Try_Jobdir</code> form, but -accepts an addtional <code>port</code> argument (to specify the TCP port to -listen on) and a <code>userpass</code> list of username/password pairs to -accept. Remember to use good passwords for this: the security of the -buildslave accounts depends upon it: - -<pre class="example"> from buildbot.scheduler import Try_Userpass - s = Try_Userpass("try2", ["full-linux", "full-netbsd", "full-OSX"], - port=8031, userpass=[("alice","pw1"), ("bob", "pw2")] ) - c['schedulers'] = [s] -</pre> - <p>Like most places in the buildbot, the <code>port</code> argument takes a -strports specification. See <code>twisted.application.strports</code> for -details. - -<h3 class="heading">locating the master</h3> - -<p>The <samp><span class="command">try</span></samp> command needs to be told how to connect to the -<code>TryScheduler</code>, and must know which of the authentication -approaches described above is in use by the buildmaster. You specify -the approach by using <samp><span class="option">--connect=ssh</span></samp> or <samp><span class="option">--connect=pb</span></samp> -(or <code>try_connect = 'ssh'</code> or <code>try_connect = 'pb'</code> in -<samp><span class="file">.buildbot/options</span></samp>). - - <p>For the PB approach, the command must be given a <samp><span class="option">--master</span></samp> -argument (in the form HOST:PORT) that points to TCP port that you -picked in the <code>Try_Userpass</code> scheduler. It also takes a -<samp><span class="option">--username</span></samp> and <samp><span class="option">--passwd</span></samp> pair of arguments that match -one of the entries in the buildmaster's <code>userpass</code> list. These -arguments can also be provided as <code>try_master</code>, -<code>try_username</code>, and <code>try_password</code> entries in the -<samp><span class="file">.buildbot/options</span></samp> file. - - <p>For the SSH approach, the command must be given <samp><span class="option">--tryhost</span></samp>, -<samp><span class="option">--username</span></samp>, and optionally <samp><span class="option">--password</span></samp> (TODO: -really?) to get to the buildmaster host. It must also be given -<samp><span class="option">--trydir</span></samp>, which points to the inlet directory configured -above. The trydir can be relative to the user's home directory, but -most of the time you will use an explicit path like -<samp><span class="file">~buildbot/project/trydir</span></samp>. These arguments can be provided in -<samp><span class="file">.buildbot/options</span></samp> as <code>try_host</code>, <code>try_username</code>, -<code>try_password</code>, and <code>try_dir</code>. - - <p>In addition, the SSH approach needs to connect to a PBListener status -port, so it can retrieve and report the results of the build (the PB -approach uses the existing connection to retrieve status information, -so this step is not necessary). This requires a <samp><span class="option">--master</span></samp> -argument, or a <code>masterstatus</code> entry in <samp><span class="file">.buildbot/options</span></samp>, -in the form of a HOSTNAME:PORT string. - -<h3 class="heading">choosing the Builders</h3> - -<p>A trial build is performed on multiple Builders at the same time, and -the developer gets to choose which Builders are used (limited to a set -selected by the buildmaster admin with the TryScheduler's -<code>builderNames=</code> argument). The set you choose will depend upon -what your goals are: if you are concerned about cross-platform -compatibility, you should use multiple Builders, one from each -platform of interest. You might use just one builder if that platform -has libraries or other facilities that allow better test coverage than -what you can accomplish on your own machine, or faster test runs. - - <p>The set of Builders to use can be specified with multiple -<samp><span class="option">--builder</span></samp> arguments on the command line. It can also be -specified with a single <code>try_builders</code> option in -<samp><span class="file">.buildbot/options</span></samp> that uses a list of strings to specify all -the Builder names: - -<pre class="example"> try_builders = ["full-OSX", "full-win32", "full-linux"] -</pre> - <h3 class="heading">specifying the VC system</h3> - -<p>The <samp><span class="command">try</span></samp> command also needs to know how to take the -developer's current tree and extract the (revision, patch) -source-stamp pair. Each VC system uses a different process, so you -start by telling the <samp><span class="command">try</span></samp> command which VC system you are -using, with an argument like <samp><span class="option">--vc=cvs</span></samp> or <samp><span class="option">--vc=tla</span></samp>. -This can also be provided as <code>try_vc</code> in -<samp><span class="file">.buildbot/options</span></samp>. - - <p>The following names are recognized: <code>cvs</code> <code>svn</code> <code>baz</code> -<code>tla</code> <code>hg</code> <code>darcs</code> - -<h3 class="heading">finding the top of the tree</h3> - -<p>Some VC systems (notably CVS and SVN) track each directory -more-or-less independently, which means the <samp><span class="command">try</span></samp> command -needs to move up to the top of the project tree before it will be able -to construct a proper full-tree patch. To accomplish this, the -<samp><span class="command">try</span></samp> command will crawl up through the parent directories -until it finds a marker file. The default name for this marker file is -<samp><span class="file">.buildbot-top</span></samp>, so when you are using CVS or SVN you should -<code>touch .buildbot-top</code> from the top of your tree before running -<samp><span class="command">buildbot try</span></samp>. Alternatively, you can use a filename like -<samp><span class="file">ChangeLog</span></samp> or <samp><span class="file">README</span></samp>, since many projects put one of -these files in their top-most directory (and nowhere else). To set -this filename, use <samp><span class="option">--try-topfile=ChangeLog</span></samp>, or set it in the -options file with <code>try_topfile = 'ChangeLog'</code>. - - <p>You can also manually set the top of the tree with -<samp><span class="option">--try-topdir=~/trees/mytree</span></samp>, or <code>try_topdir = -'~/trees/mytree'</code>. If you use <code>try_topdir</code>, in a -<samp><span class="file">.buildbot/options</span></samp> file, you will need a separate options file -for each tree you use, so it may be more convenient to use the -<code>try_topfile</code> approach instead. - - <p>Other VC systems which work on full projects instead of individual -directories (tla, baz, darcs, monotone, mercurial, git) do not require -<samp><span class="command">try</span></samp> to know the top directory, so the <samp><span class="option">--try-topfile</span></samp> -and <samp><span class="option">--try-topdir</span></samp> arguments will be ignored. -<!-- is this true? I think I currently require topdirs all the time. --> - - <p>If the <samp><span class="command">try</span></samp> command cannot find the top directory, it will -abort with an error message. - -<h3 class="heading">determining the branch name</h3> - -<p>Some VC systems record the branch information in a way that “try” -can locate it, in particular Arch (both <samp><span class="command">tla</span></samp> and -<samp><span class="command">baz</span></samp>). For the others, if you are using something other than -the default branch, you will have to tell the buildbot which branch -your tree is using. You can do this with either the <samp><span class="option">--branch</span></samp> -argument, or a <samp><span class="option">try_branch</span></samp> entry in the -<samp><span class="file">.buildbot/options</span></samp> file. - -<h3 class="heading">determining the revision and patch</h3> - -<p>Each VC system has a separate approach for determining the tree's base -revision and computing a patch. - - <dl> -<dt><code>CVS</code><dd> -<samp><span class="command">try</span></samp> pretends that the tree is up to date. It converts the -current time into a <code>-D</code> time specification, uses it as the base -revision, and computes the diff between the upstream tree as of that -point in time versus the current contents. This works, more or less, -but requires that the local clock be in reasonably good sync with the -repository. - - <br><dt><code>SVN</code><dd><samp><span class="command">try</span></samp> does a <code>svn status -u</code> to find the latest -repository revision number (emitted on the last line in the “Status -against revision: NN” message). It then performs an <code>svn diff --rNN</code> to find out how your tree differs from the repository version, -and sends the resulting patch to the buildmaster. If your tree is not -up to date, this will result in the “try” tree being created with -the latest revision, then <em>backwards</em> patches applied to bring it -“back” to the version you actually checked out (plus your actual -code changes), but this will still result in the correct tree being -used for the build. - - <br><dt><code>baz</code><dd><samp><span class="command">try</span></samp> does a <code>baz tree-id</code> to determine the -fully-qualified version and patch identifier for the tree -(ARCHIVE/VERSION–patch-NN), and uses the VERSION–patch-NN component -as the base revision. It then does a <code>baz diff</code> to obtain the -patch. - - <br><dt><code>tla</code><dd><samp><span class="command">try</span></samp> does a <code>tla tree-version</code> to get the -fully-qualified version identifier (ARCHIVE/VERSION), then takes the -first line of <code>tla logs --reverse</code> to figure out the base -revision. Then it does <code>tla changes --diffs</code> to obtain the patch. - - <br><dt><code>Darcs</code><dd><code>darcs changes --context</code> emits a text file that contains a list -of all patches back to and including the last tag was made. This text -file (plus the location of a repository that contains all these -patches) is sufficient to re-create the tree. Therefore the contents -of this “context” file <em>are</em> the revision stamp for a -Darcs-controlled source tree. - - <p>So <samp><span class="command">try</span></samp> does a <code>darcs changes --context</code> to determine -what your tree's base revision is, and then does a <code>darcs diff --u</code> to compute the patch relative to that revision. - - <br><dt><code>Mercurial</code><dd><code>hg identify</code> emits a short revision ID (basically a truncated -SHA1 hash of the current revision's contents), which is used as the -base revision. <code>hg diff</code> then provides the patch relative to that -revision. For <samp><span class="command">try</span></samp> to work, your working directory must only -have patches that are available from the same remotely-available -repository that the build process' <code>step.Mercurial</code> will use. - - <br><dt><code>Git</code><dd><code>git branch -v</code> lists all the branches available in the local -repository along with the revision ID it points to and a short summary -of the last commit. The line containing the currently checked out -branch begins with '* ' (star and space) while all the others start -with ' ' (two spaces). <samp><span class="command">try</span></samp> scans for this line and extracts -the branch name and revision from it. Then it generates a diff against -the base revision. -<!-- TODO: I'm not sure if this actually works the way it's intended --> -<!-- since the extracted base revision might not actually exist in the --> -<!-- upstream repository. Perhaps we need to add a -remote option to --> -<!-- specify the remote tracking branch to generate a diff against. --> - - <!-- TODO: monotone --> - </dl> - -<h3 class="heading">waiting for results</h3> - -<p>If you provide the <samp><span class="option">--wait</span></samp> option (or <code>try_wait = True</code> -in <samp><span class="file">.buildbot/options</span></samp>), the <samp><span class="command">buildbot try</span></samp> command will -wait until your changes have either been proven good or bad before -exiting. Unless you use the <samp><span class="option">--quiet</span></samp> option (or -<code>try_quiet=True</code>), it will emit a progress message every 60 -seconds until the builds have completed. - -<ul class="menu"> -<li><a accesskey="1" href="#try-_002d_002ddiff">try --diff</a> -</ul> - -<div class="node"> -<p><hr> -<a name="try---diff"></a> -<a name="try-_002d_002ddiff"></a> -Previous: <a rel="previous" accesskey="p" href="#try">try</a>, -Up: <a rel="up" accesskey="u" href="#try">try</a> - -</div> - -<h5 class="subsubsection">8.2.3.1 try –diff</h5> - -<p>Sometimes you might have a patch from someone else that you want to -submit to the buildbot. For example, a user may have created a patch -to fix some specific bug and sent it to you by email. You've inspected -the patch and suspect that it might do the job (and have at least -confirmed that it doesn't do anything evil). Now you want to test it -out. - - <p>One approach would be to check out a new local tree, apply the patch, -run your local tests, then use “buildbot try” to run the tests on -other platforms. An alternate approach is to use the <samp><span class="command">buildbot -try --diff</span></samp> form to have the buildbot test the patch without using a -local tree. - - <p>This form takes a <samp><span class="option">--diff</span></samp> argument which points to a file that -contains the patch you want to apply. By default this patch will be -applied to the TRUNK revision, but if you give the optional -<samp><span class="option">--baserev</span></samp> argument, a tree of the given revision will be used -as a starting point instead of TRUNK. - - <p>You can also use <samp><span class="command">buildbot try --diff=-</span></samp> to read the patch -from stdin. - - <p>Each patch has a “patchlevel” associated with it. This indicates the -number of slashes (and preceding pathnames) that should be stripped -before applying the diff. This exactly corresponds to the <samp><span class="option">-p</span></samp> -or <samp><span class="option">--strip</span></samp> argument to the <samp><span class="command">patch</span></samp> utility. By -default <samp><span class="command">buildbot try --diff</span></samp> uses a patchlevel of 0, but you -can override this with the <samp><span class="option">-p</span></samp> argument. - - <p>When you use <samp><span class="option">--diff</span></samp>, you do not need to use any of the other -options that relate to a local tree, specifically <samp><span class="option">--vc</span></samp>, -<samp><span class="option">--try-topfile</span></samp>, or <samp><span class="option">--try-topdir</span></samp>. These options will -be ignored. Of course you must still specify how to get to the -buildmaster (with <samp><span class="option">--connect</span></samp>, <samp><span class="option">--tryhost</span></samp>, etc). - -<div class="node"> -<p><hr> -<a name="Other-Tools"></a> -Next: <a rel="next" accesskey="n" href="#g_t_002ebuildbot-config-directory">.buildbot config directory</a>, -Previous: <a rel="previous" accesskey="p" href="#Developer-Tools">Developer Tools</a>, -Up: <a rel="up" accesskey="u" href="#Command_002dline-tool">Command-line tool</a> - -</div> - -<h3 class="section">8.3 Other Tools</h3> - -<p>These tools are generally used by buildmaster administrators. - -<ul class="menu"> -<li><a accesskey="1" href="#sendchange">sendchange</a> -<li><a accesskey="2" href="#debugclient">debugclient</a> -</ul> - -<div class="node"> -<p><hr> -<a name="sendchange"></a> -Next: <a rel="next" accesskey="n" href="#debugclient">debugclient</a>, -Previous: <a rel="previous" accesskey="p" href="#Other-Tools">Other Tools</a>, -Up: <a rel="up" accesskey="u" href="#Other-Tools">Other Tools</a> - -</div> - -<h4 class="subsection">8.3.1 sendchange</h4> - -<p>This command is used to tell the buildmaster about source changes. It -is intended to be used from within a commit script, installed on the -VC server. It requires that you have a PBChangeSource -(see <a href="#PBChangeSource">PBChangeSource</a>) running in the buildmaster (by being set in -<code>c['change_source']</code>). - -<pre class="example"> buildbot sendchange --master <var>MASTERHOST</var>:<var>PORT</var> --username <var>USER</var> <var>FILENAMES..</var> -</pre> - <p>There are other (optional) arguments which can influence the -<code>Change</code> that gets submitted: - - <dl> -<dt><code>--branch</code><dd>This provides the (string) branch specifier. If omitted, it defaults -to None, indicating the “default branch”. All files included in this -Change must be on the same branch. - - <br><dt><code>--category</code><dd>This provides the (string) category specifier. If omitted, it defaults -to None, indicating “no category”. The category property is used -by Schedulers to filter what changes they listen to. - - <br><dt><code>--revision_number</code><dd>This provides a (numeric) revision number for the change, used for VC systems -that use numeric transaction numbers (like Subversion). - - <br><dt><code>--revision</code><dd>This provides a (string) revision specifier, for VC systems that use -strings (Arch would use something like patch-42 etc). - - <br><dt><code>--revision_file</code><dd>This provides a filename which will be opened and the contents used as -the revision specifier. This is specifically for Darcs, which uses the -output of <samp><span class="command">darcs changes --context</span></samp> as a revision specifier. -This context file can be a couple of kilobytes long, spanning a couple -lines per patch, and would be a hassle to pass as a command-line -argument. - - <br><dt><code>--comments</code><dd>This provides the change comments as a single argument. You may want -to use <samp><span class="option">--logfile</span></samp> instead. - - <br><dt><code>--logfile</code><dd>This instructs the tool to read the change comments from the given -file. If you use <code>-</code> as the filename, the tool will read the -change comments from stdin. -</dl> - -<div class="node"> -<p><hr> -<a name="debugclient"></a> -Previous: <a rel="previous" accesskey="p" href="#sendchange">sendchange</a>, -Up: <a rel="up" accesskey="u" href="#Other-Tools">Other Tools</a> - -</div> - -<h4 class="subsection">8.3.2 debugclient</h4> - -<pre class="example"> buildbot debugclient --master <var>MASTERHOST</var>:<var>PORT</var> --passwd <var>DEBUGPW</var> -</pre> - <p>This launches a small Gtk+/Glade-based debug tool, connecting to the -buildmaster's “debug port”. This debug port shares the same port -number as the slaveport (see <a href="#Setting-the-slaveport">Setting the slaveport</a>), but the -<code>debugPort</code> is only enabled if you set a debug password in the -buildmaster's config file (see <a href="#Debug-options">Debug options</a>). The -<samp><span class="option">--passwd</span></samp> option must match the <code>c['debugPassword']</code> -value. - - <p><samp><span class="option">--master</span></samp> can also be provided in <samp><span class="file">.debug/options</span></samp> by the -<code>master</code> key. <samp><span class="option">--passwd</span></samp> can be provided by the -<code>debugPassword</code> key. - - <p>The <code>Connect</code> button must be pressed before any of the other -buttons will be active. This establishes the connection to the -buildmaster. The other sections of the tool are as follows: - - <dl> -<dt><code>Reload .cfg</code><dd>Forces the buildmaster to reload its <samp><span class="file">master.cfg</span></samp> file. This is -equivalent to sending a SIGHUP to the buildmaster, but can be done -remotely through the debug port. Note that it is a good idea to be -watching the buildmaster's <samp><span class="file">twistd.log</span></samp> as you reload the config -file, as any errors which are detected in the config file will be -announced there. - - <br><dt><code>Rebuild .py</code><dd>(not yet implemented). The idea here is to use Twisted's “rebuild” -facilities to replace the buildmaster's running code with a new -version. Even if this worked, it would only be used by buildbot -developers. - - <br><dt><code>poke IRC</code><dd>This locates a <code>words.IRC</code> status target and causes it to emit a -message on all the channels to which it is currently connected. This -was used to debug a problem in which the buildmaster lost the -connection to the IRC server and did not attempt to reconnect. - - <br><dt><code>Commit</code><dd>This allows you to inject a Change, just as if a real one had been -delivered by whatever VC hook you are using. You can set the name of -the committed file and the name of the user who is doing the commit. -Optionally, you can also set a revision for the change. If the -revision you provide looks like a number, it will be sent as an -integer, otherwise it will be sent as a string. - - <br><dt><code>Force Build</code><dd>This lets you force a Builder (selected by name) to start a build of -the current source tree. - - <br><dt><code>Currently</code><dd>(obsolete). This was used to manually set the status of the given -Builder, but the status-assignment code was changed in an incompatible -way and these buttons are no longer meaningful. - - </dl> - -<div class="node"> -<p><hr> -<a name=".buildbot-config-directory"></a> -<a name="g_t_002ebuildbot-config-directory"></a> -Previous: <a rel="previous" accesskey="p" href="#Other-Tools">Other Tools</a>, -Up: <a rel="up" accesskey="u" href="#Command_002dline-tool">Command-line tool</a> - -</div> - -<h3 class="section">8.4 .buildbot config directory</h3> - -<p>Many of the <samp><span class="command">buildbot</span></samp> tools must be told how to contact the -buildmaster that they interact with. This specification can be -provided as a command-line argument, but most of the time it will be -easier to set them in an “options” file. The <samp><span class="command">buildbot</span></samp> -command will look for a special directory named <samp><span class="file">.buildbot</span></samp>, -starting from the current directory (where the command was run) and -crawling upwards, eventually looking in the user's home directory. It -will look for a file named <samp><span class="file">options</span></samp> in this directory, and will -evaluate it as a python script, looking for certain names to be set. -You can just put simple <code>name = 'value'</code> pairs in this file to -set the options. - - <p>For a description of the names used in this file, please see the -documentation for the individual <samp><span class="command">buildbot</span></samp> sub-commands. The -following is a brief sample of what this file's contents could be. - -<pre class="example"> # for status-reading tools - masterstatus = 'buildbot.example.org:12345' - # for 'sendchange' or the debug port - master = 'buildbot.example.org:18990' - debugPassword = 'eiv7Po' -</pre> - <dl> -<dt><code>masterstatus</code><dd>Location of the <code>client.PBListener</code> status port, used by -<samp><span class="command">statuslog</span></samp> and <samp><span class="command">statusgui</span></samp>. - - <br><dt><code>master</code><dd>Location of the <code>debugPort</code> (for <samp><span class="command">debugclient</span></samp>). Also the -location of the <code>pb.PBChangeSource</code> (for <samp><span class="command">sendchange</span></samp>). -Usually shares the slaveport, but a future version may make it -possible to have these listen on a separate port number. - - <br><dt><code>debugPassword</code><dd>Must match the value of <code>c['debugPassword']</code>, used to protect the -debug port, for the <samp><span class="command">debugclient</span></samp> command. - - <br><dt><code>username</code><dd>Provides a default username for the <samp><span class="command">sendchange</span></samp> command. - - </dl> - - <p>The following options are used by the <code>buildbot try</code> command -(see <a href="#try">try</a>): - - <dl> -<dt><code>try_connect</code><dd>This specifies how the “try” command should deliver its request to -the buildmaster. The currently accepted values are “ssh” and “pb”. -<br><dt><code>try_builders</code><dd>Which builders should be used for the “try” build. -<br><dt><code>try_vc</code><dd>This specifies the version control system being used. -<br><dt><code>try_branch</code><dd>This indicates that the current tree is on a non-trunk branch. -<br><dt><code>try_topdir</code><br><dt><code>try_topfile</code><dd>Use <code>try_topdir</code> to explicitly indicate the top of your working -tree, or <code>try_topfile</code> to name a file that will only be found in -that top-most directory. - - <br><dt><code>try_host</code><br><dt><code>try_username</code><br><dt><code>try_dir</code><dd>When try_connect is “ssh”, the command will pay attention to -<code>try_host</code>, <code>try_username</code>, and <code>try_dir</code>. - - <br><dt><code>try_username</code><br><dt><code>try_password</code><br><dt><code>try_master</code><dd>Instead, when <code>try_connect</code> is “pb”, the command will pay -attention to <code>try_username</code>, <code>try_password</code>, and -<code>try_master</code>. - - <br><dt><code>try_wait</code><br><dt><code>masterstatus</code><dd><code>try_wait</code> and <code>masterstatus</code> are used to ask the “try” -command to wait for the requested build to complete. - - </dl> - -<div class="node"> -<p><hr> -<a name="Resources"></a> -Next: <a rel="next" accesskey="n" href="#Developer_0027s-Appendix">Developer's Appendix</a>, -Previous: <a rel="previous" accesskey="p" href="#Command_002dline-tool">Command-line tool</a>, -Up: <a rel="up" accesskey="u" href="#Top">Top</a> - -</div> - -<h2 class="chapter">9 Resources</h2> - -<p>The Buildbot's home page is at <a href="http://buildbot.sourceforge.net/">http://buildbot.sourceforge.net/</a> - - <p>For configuration questions and general discussion, please use the -<code>buildbot-devel</code> mailing list. The subscription instructions and -archives are available at -<a href="http://lists.sourceforge.net/lists/listinfo/buildbot-devel">http://lists.sourceforge.net/lists/listinfo/buildbot-devel</a> - -<div class="node"> -<p><hr> -<a name="Developer's-Appendix"></a> -<a name="Developer_0027s-Appendix"></a> -Next: <a rel="next" accesskey="n" href="#Index-of-Useful-Classes">Index of Useful Classes</a>, -Previous: <a rel="previous" accesskey="p" href="#Resources">Resources</a>, -Up: <a rel="up" accesskey="u" href="#Top">Top</a> - -</div> - -<h2 class="unnumbered">Developer's Appendix</h2> - -<p>This appendix contains random notes about the implementation of the -Buildbot, and is likely to only be of use to people intending to -extend the Buildbot's internals. - - <p>The buildmaster consists of a tree of Service objects, which is shaped -as follows: - -<pre class="example"> BuildMaster - ChangeMaster (in .change_svc) - [IChangeSource instances] - [IScheduler instances] (in .schedulers) - BotMaster (in .botmaster) - [IBuildSlave instances] - [IStatusTarget instances] (in .statusTargets) -</pre> - <p>The BotMaster has a collection of Builder objects as values of its -<code>.builders</code> dictionary. - -<div class="node"> -<p><hr> -<a name="Index-of-Useful-Classes"></a> -Next: <a rel="next" accesskey="n" href="#Index-of-master_002ecfg-keys">Index of master.cfg keys</a>, -Previous: <a rel="previous" accesskey="p" href="#Developer_0027s-Appendix">Developer's Appendix</a>, -Up: <a rel="up" accesskey="u" href="#Top">Top</a> - -</div> - -<h2 class="unnumbered">Index of Useful Classes</h2> - -<p>This is a list of all user-visible classes. There are the ones that -are useful in <samp><span class="file">master.cfg</span></samp>, the buildmaster's configuration file. -Classes that are not listed here are generally internal things that -admins are unlikely to have much use for. - -<h3 class="heading">Change Sources</h3> - -<ul class="index-cs" compact> -<li><a href="#index-buildbot_002echanges_002ebonsaipoller_002eBonsaiPoller-49"><code>buildbot.changes.bonsaipoller.BonsaiPoller</code></a>: <a href="#BonsaiPoller">BonsaiPoller</a></li> -<li><a href="#index-buildbot_002echanges_002efreshcvs_002eFreshCVSSource-42"><code>buildbot.changes.freshcvs.FreshCVSSource</code></a>: <a href="#CVSToys-_002d-PBService">CVSToys - PBService</a></li> -<li><a href="#index-buildbot_002echanges_002email_002eBonsaiMaildirSource-45"><code>buildbot.changes.mail.BonsaiMaildirSource</code></a>: <a href="#BonsaiMaildirSource">BonsaiMaildirSource</a></li> -<li><a href="#index-buildbot_002echanges_002email_002eFCMaildirSource-43"><code>buildbot.changes.mail.FCMaildirSource</code></a>: <a href="#FCMaildirSource">FCMaildirSource</a></li> -<li><a href="#index-buildbot_002echanges_002email_002eSVNCommitEmailMaildirSource-46"><code>buildbot.changes.mail.SVNCommitEmailMaildirSource</code></a>: <a href="#SVNCommitEmailMaildirSource">SVNCommitEmailMaildirSource</a></li> -<li><a href="#index-buildbot_002echanges_002email_002eSyncmailMaildirSource-44"><code>buildbot.changes.mail.SyncmailMaildirSource</code></a>: <a href="#SyncmailMaildirSource">SyncmailMaildirSource</a></li> -<li><a href="#index-buildbot_002echanges_002ep4poller_002eP4Source-48"><code>buildbot.changes.p4poller.P4Source</code></a>: <a href="#P4Source">P4Source</a></li> -<li><a href="#index-buildbot_002echanges_002epb_002ePBChangeSource-47"><code>buildbot.changes.pb.PBChangeSource</code></a>: <a href="#PBChangeSource">PBChangeSource</a></li> -<li><a href="#index-buildbot_002echanges_002esvnpoller_002eSVNPoller-50"><code>buildbot.changes.svnpoller.SVNPoller</code></a>: <a href="#SVNPoller">SVNPoller</a></li> - </ul><h3 class="heading">Schedulers and Locks</h3> - - - -<ul class="index-sl" compact> -<li><a href="#index-buildbot_002elocks_002eLockAccess-93"><code>buildbot.locks.LockAccess</code></a>: <a href="#Interlocks">Interlocks</a></li> -<li><a href="#index-buildbot_002elocks_002eMasterLock-91"><code>buildbot.locks.MasterLock</code></a>: <a href="#Interlocks">Interlocks</a></li> -<li><a href="#index-buildbot_002elocks_002eSlaveLock-92"><code>buildbot.locks.SlaveLock</code></a>: <a href="#Interlocks">Interlocks</a></li> -<li><a href="#index-buildbot_002escheduler_002eAnyBranchScheduler-22"><code>buildbot.scheduler.AnyBranchScheduler</code></a>: <a href="#AnyBranchScheduler">AnyBranchScheduler</a></li> -<li><a href="#index-buildbot_002escheduler_002eDependent-25"><code>buildbot.scheduler.Dependent</code></a>: <a href="#Dependent-Scheduler">Dependent Scheduler</a></li> -<li><a href="#index-buildbot_002escheduler_002eNightly-27"><code>buildbot.scheduler.Nightly</code></a>: <a href="#Nightly-Scheduler">Nightly Scheduler</a></li> -<li><a href="#index-buildbot_002escheduler_002ePeriodic-26"><code>buildbot.scheduler.Periodic</code></a>: <a href="#Periodic-Scheduler">Periodic Scheduler</a></li> -<li><a href="#index-buildbot_002escheduler_002eScheduler-21"><code>buildbot.scheduler.Scheduler</code></a>: <a href="#Scheduler-Scheduler">Scheduler Scheduler</a></li> -<li><a href="#index-buildbot_002escheduler_002eTriggerable-31"><code>buildbot.scheduler.Triggerable</code></a>: <a href="#Triggerable-Scheduler">Triggerable Scheduler</a></li> -<li><a href="#index-buildbot_002escheduler_002eTry_005fJobdir-120"><code>buildbot.scheduler.Try_Jobdir</code></a>: <a href="#try">try</a></li> -<li><a href="#index-buildbot_002escheduler_002eTry_005fJobdir-28"><code>buildbot.scheduler.Try_Jobdir</code></a>: <a href="#Try-Schedulers">Try Schedulers</a></li> -<li><a href="#index-buildbot_002escheduler_002eTry_005fUserpass-121"><code>buildbot.scheduler.Try_Userpass</code></a>: <a href="#try">try</a></li> -<li><a href="#index-buildbot_002escheduler_002eTry_005fUserpass-29"><code>buildbot.scheduler.Try_Userpass</code></a>: <a href="#Try-Schedulers">Try Schedulers</a></li> - </ul><h3 class="heading">Build Factories</h3> - - - -<ul class="index-bf" compact> -<li><a href="#index-buildbot_002eprocess_002efactory_002eBasicBuildFactory-95"><code>buildbot.process.factory.BasicBuildFactory</code></a>: <a href="#BuildFactory">BuildFactory</a></li> -<li><a href="#index-buildbot_002eprocess_002efactory_002eBasicSVN-96"><code>buildbot.process.factory.BasicSVN</code></a>: <a href="#BuildFactory">BuildFactory</a></li> -<li><a href="#index-buildbot_002eprocess_002efactory_002eBuildFactory-94"><code>buildbot.process.factory.BuildFactory</code></a>: <a href="#BuildFactory">BuildFactory</a></li> -<li><a href="#index-buildbot_002eprocess_002efactory_002eCPAN-100"><code>buildbot.process.factory.CPAN</code></a>: <a href="#CPAN">CPAN</a></li> -<li><a href="#index-buildbot_002eprocess_002efactory_002eDistutils-101"><code>buildbot.process.factory.Distutils</code></a>: <a href="#Python-distutils">Python distutils</a></li> -<li><a href="#index-buildbot_002eprocess_002efactory_002eGNUAutoconf-99"><code>buildbot.process.factory.GNUAutoconf</code></a>: <a href="#GNUAutoconf">GNUAutoconf</a></li> -<li><a href="#index-buildbot_002eprocess_002efactory_002eQuickBuildFactory-98"><code>buildbot.process.factory.QuickBuildFactory</code></a>: <a href="#Quick-builds">Quick builds</a></li> -<li><a href="#index-buildbot_002eprocess_002efactory_002eTrial-102"><code>buildbot.process.factory.Trial</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li> - </ul><h3 class="heading">Build Steps</h3> - - - -<ul class="index-bs" compact> -<li><a href="#index-buildbot_002esteps_002emaxq_002eMaxQ-123"><code>buildbot.steps.maxq.MaxQ</code></a>: <a href="#Index-of-Useful-Classes">Index of Useful Classes</a></li> -<li><a href="#index-buildbot_002esteps_002epython_002eBuildEPYDoc-78"><code>buildbot.steps.python.BuildEPYDoc</code></a>: <a href="#BuildEPYDoc">BuildEPYDoc</a></li> -<li><a href="#index-buildbot_002esteps_002epython_002ePyFlakes-79"><code>buildbot.steps.python.PyFlakes</code></a>: <a href="#PyFlakes">PyFlakes</a></li> -<li><a href="#index-buildbot_002esteps_002epython_002ePyLint-80"><code>buildbot.steps.python.PyLint</code></a>: <a href="#PyLint">PyLint</a></li> -<li><a href="#index-buildbot_002esteps_002epython_005ftwisted_002eBuildDebs-106"><code>buildbot.steps.python_twisted.BuildDebs</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li> -<li><a href="#index-buildbot_002esteps_002epython_005ftwisted_002eHLint-103"><code>buildbot.steps.python_twisted.HLint</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li> -<li><a href="#index-buildbot_002esteps_002epython_005ftwisted_002eProcessDocs-105"><code>buildbot.steps.python_twisted.ProcessDocs</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li> -<li><a href="#index-buildbot_002esteps_002epython_005ftwisted_002eRemovePYCs-107"><code>buildbot.steps.python_twisted.RemovePYCs</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li> -<li><a href="#index-buildbot_002esteps_002epython_005ftwisted_002eTrial-104"><code>buildbot.steps.python_twisted.Trial</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li> -<li><a href="#index-buildbot_002esteps_002eshell_002eCompile-73"><code>buildbot.steps.shell.Compile</code></a>: <a href="#Compile">Compile</a></li> -<li><a href="#index-buildbot_002esteps_002eshell_002eConfigure-72"><code>buildbot.steps.shell.Configure</code></a>: <a href="#Configure">Configure</a></li> -<li><a href="#index-buildbot_002esteps_002eshell_002ePerlModuleTest-76"><code>buildbot.steps.shell.PerlModuleTest</code></a>: <a href="#PerlModuleTest">PerlModuleTest</a></li> -<li><a href="#index-buildbot_002esteps_002eshell_002eSetProperty-77"><code>buildbot.steps.shell.SetProperty</code></a>: <a href="#SetProperty">SetProperty</a></li> -<li><a href="#index-buildbot_002esteps_002eshell_002eShellCommand-71"><code>buildbot.steps.shell.ShellCommand</code></a>: <a href="#ShellCommand">ShellCommand</a></li> -<li><a href="#index-buildbot_002esteps_002eshell_002eTest-74"><code>buildbot.steps.shell.Test</code></a>: <a href="#Test">Test</a></li> -<li><a href="#index-buildbot_002esteps_002eshell_002eTreeSize-75"><code>buildbot.steps.shell.TreeSize</code></a>: <a href="#TreeSize">TreeSize</a></li> -<li><a href="#index-buildbot_002esteps_002esource_002eArch-62"><code>buildbot.steps.source.Arch</code></a>: <a href="#Arch">Arch</a></li> -<li><a href="#index-buildbot_002esteps_002esource_002eBazaar-64"><code>buildbot.steps.source.Bazaar</code></a>: <a href="#Bazaar">Bazaar</a></li> -<li><a href="#index-buildbot_002esteps_002esource_002eBzr-66"><code>buildbot.steps.source.Bzr</code></a>: <a href="#Bzr">Bzr</a></li> -<li><a href="#index-buildbot_002esteps_002esource_002eCVS-54"><code>buildbot.steps.source.CVS</code></a>: <a href="#CVS">CVS</a></li> -<li><a href="#index-buildbot_002esteps_002esource_002eDarcs-58"><code>buildbot.steps.source.Darcs</code></a>: <a href="#Darcs">Darcs</a></li> -<li><a href="#index-buildbot_002esteps_002esource_002eGit-122"><code>buildbot.steps.source.Git</code></a>: <a href="#Index-of-Useful-Classes">Index of Useful Classes</a></li> -<li><a href="#index-buildbot_002esteps_002esource_002eGit-70"><code>buildbot.steps.source.Git</code></a>: <a href="#Git">Git</a></li> -<li><a href="#index-buildbot_002esteps_002esource_002eMercurial-60"><code>buildbot.steps.source.Mercurial</code></a>: <a href="#Mercurial">Mercurial</a></li> -<li><a href="#index-buildbot_002esteps_002esource_002eP4-68"><code>buildbot.steps.source.P4</code></a>: <a href="#P4">P4</a></li> -<li><a href="#index-buildbot_002esteps_002esource_002eSVN-56"><code>buildbot.steps.source.SVN</code></a>: <a href="#SVN">SVN</a></li> -<li><a href="#index-buildbot_002esteps_002etransfer_002eDirectoryUpload-84"><code>buildbot.steps.transfer.DirectoryUpload</code></a>: <a href="#Transferring-Files">Transferring Files</a></li> -<li><a href="#index-buildbot_002esteps_002etransfer_002eFileDownload-83"><code>buildbot.steps.transfer.FileDownload</code></a>: <a href="#Transferring-Files">Transferring Files</a></li> -<li><a href="#index-buildbot_002esteps_002etransfer_002eFileUpload-82"><code>buildbot.steps.transfer.FileUpload</code></a>: <a href="#Transferring-Files">Transferring Files</a></li> - </ul><!-- undocumented steps --> -<p><a name="index-buildbot_002esteps_002esource_002eGit-122"></a><a name="index-buildbot_002esteps_002emaxq_002eMaxQ-123"></a> - -<h3 class="heading">Status Targets</h3> - - - -<ul class="index-st" compact> -<li><a href="#index-buildbot_002estatus_002eclient_002ePBListener-118"><code>buildbot.status.client.PBListener</code></a>: <a href="#PBListener">PBListener</a></li> -<li><a href="#index-buildbot_002estatus_002ehtml_002eWaterfall-111"><code>buildbot.status.html.Waterfall</code></a>: <a href="#HTML-Waterfall">HTML Waterfall</a></li> -<li><a href="#index-buildbot_002estatus_002email_002eMailNotifier-114"><code>buildbot.status.mail.MailNotifier</code></a>: <a href="#MailNotifier">MailNotifier</a></li> -<li><a href="#index-buildbot_002estatus_002eweb_002ebaseweb_002eWebStatus-109"><code>buildbot.status.web.baseweb.WebStatus</code></a>: <a href="#WebStatus">WebStatus</a></li> -<li><a href="#index-buildbot_002estatus_002ewords_002eIRC-116"><code>buildbot.status.words.IRC</code></a>: <a href="#IRC-Bot">IRC Bot</a></li> - </ul><!-- TODO: undocumented targets --> -<div class="node"> -<p><hr> -<a name="Index-of-master.cfg-keys"></a> -<a name="Index-of-master_002ecfg-keys"></a> -Next: <a rel="next" accesskey="n" href="#Index">Index</a>, -Previous: <a rel="previous" accesskey="p" href="#Index-of-Useful-Classes">Index of Useful Classes</a>, -Up: <a rel="up" accesskey="u" href="#Top">Top</a> - -</div> - -<h2 class="unnumbered">Index of master.cfg keys</h2> - -<p>This is a list of all of the significant keys in master.cfg . Recall -that master.cfg is effectively a small python program with exactly one -responsibility: create a dictionary named <code>BuildmasterConfig</code>. -The keys of this dictionary are listed here. The beginning of the -master.cfg file typically starts with something like: - -<pre class="example"> BuildmasterConfig = c = {} -</pre> - <p>Therefore a config key of <code>change_source</code> will usually appear in -master.cfg as <code>c['change_source']</code>. - - - -<ul class="index-bc" compact> -<li><a href="#index-c_005b_0027buildbotURL_0027_005d-15"><code>c['buildbotURL']</code></a>: <a href="#Defining-the-Project">Defining the Project</a></li> -<li><a href="#index-c_005b_0027builders_0027_005d-38"><code>c['builders']</code></a>: <a href="#Defining-Builders">Defining Builders</a></li> -<li><a href="#index-c_005b_0027change_005fsource_0027_005d-18"><code>c['change_source']</code></a>: <a href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a></li> -<li><a href="#index-c_005b_0027debugPassword_0027_005d-40"><code>c['debugPassword']</code></a>: <a href="#Debug-options">Debug options</a></li> -<li><a href="#index-c_005b_0027logCompressionLimit_0027_005d-16"><code>c['logCompressionLimit']</code></a>: <a href="#Defining-the-Project">Defining the Project</a></li> -<li><a href="#index-c_005b_0027manhole_0027_005d-41"><code>c['manhole']</code></a>: <a href="#Debug-options">Debug options</a></li> -<li><a href="#index-c_005b_0027mergeRequests_0027_005d-32"><code>c['mergeRequests']</code></a>: <a href="#Merging-BuildRequests">Merging BuildRequests</a></li> -<li><a href="#index-c_005b_0027projectName_0027_005d-13"><code>c['projectName']</code></a>: <a href="#Defining-the-Project">Defining the Project</a></li> -<li><a href="#index-c_005b_0027projectURL_0027_005d-14"><code>c['projectURL']</code></a>: <a href="#Defining-the-Project">Defining the Project</a></li> -<li><a href="#index-c_005b_0027properties_0027_005d-36"><code>c['properties']</code></a>: <a href="#Defining-Global-Properties">Defining Global Properties</a></li> -<li><a href="#index-c_005b_0027schedulers_0027_005d-19"><code>c['schedulers']</code></a>: <a href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a></li> -<li><a href="#index-c_005b_0027slavePortnum_0027_005d-33"><code>c['slavePortnum']</code></a>: <a href="#Setting-the-slaveport">Setting the slaveport</a></li> -<li><a href="#index-c_005b_0027slaves_0027_005d-34"><code>c['slaves']</code></a>: <a href="#Buildslave-Specifiers">Buildslave Specifiers</a></li> -<li><a href="#index-c_005b_0027sources_0027_005d-17"><code>c['sources']</code></a>: <a href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a></li> -<li><a href="#index-c_005b_0027status_0027_005d-39"><code>c['status']</code></a>: <a href="#Defining-Status-Targets">Defining Status Targets</a></li> - </ul><div class="node"> -<p><hr> -<a name="Index"></a> -Previous: <a rel="previous" accesskey="p" href="#Index-of-master_002ecfg-keys">Index of master.cfg keys</a>, -Up: <a rel="up" accesskey="u" href="#Top">Top</a> - -</div> - -<h2 class="unnumbered">Index</h2> - - - -<ul class="index-cp" compact> -<li><a href="#index-addURL-89">addURL</a>: <a href="#BuildStep-URLs">BuildStep URLs</a></li> -<li><a href="#index-Arch-Checkout-61">Arch Checkout</a>: <a href="#Arch">Arch</a></li> -<li><a href="#index-Bazaar-Checkout-63">Bazaar Checkout</a>: <a href="#Bazaar">Bazaar</a></li> -<li><a href="#index-Builder-9">Builder</a>: <a href="#Builder">Builder</a></li> -<li><a href="#index-BuildRequest-8">BuildRequest</a>: <a href="#BuildRequest">BuildRequest</a></li> -<li><a href="#index-BuildSet-7">BuildSet</a>: <a href="#BuildSet">BuildSet</a></li> -<li><a href="#index-BuildStep-URLs-88">BuildStep URLs</a>: <a href="#BuildStep-URLs">BuildStep URLs</a></li> -<li><a href="#index-Bzr-Checkout-65">Bzr Checkout</a>: <a href="#Bzr">Bzr</a></li> -<li><a href="#index-Configuration-12">Configuration</a>: <a href="#Configuration">Configuration</a></li> -<li><a href="#index-CVS-Checkout-53">CVS Checkout</a>: <a href="#CVS">CVS</a></li> -<li><a href="#index-Darcs-Checkout-57">Darcs Checkout</a>: <a href="#Darcs">Darcs</a></li> -<li><a href="#index-Dependencies-24">Dependencies</a>: <a href="#Dependent-Scheduler">Dependent Scheduler</a></li> -<li><a href="#index-Dependent-23">Dependent</a>: <a href="#Dependent-Scheduler">Dependent Scheduler</a></li> -<li><a href="#index-email-112">email</a>: <a href="#MailNotifier">MailNotifier</a></li> -<li><a href="#index-File-Transfer-81">File Transfer</a>: <a href="#Transferring-Files">Transferring Files</a></li> -<li><a href="#index-Git-Checkout-69">Git Checkout</a>: <a href="#Git">Git</a></li> -<li><a href="#index-installation-3">installation</a>: <a href="#Installing-the-code">Installing the code</a></li> -<li><a href="#index-introduction-1">introduction</a>: <a href="#Introduction">Introduction</a></li> -<li><a href="#index-IRC-115">IRC</a>: <a href="#IRC-Bot">IRC Bot</a></li> -<li><a href="#index-links-87">links</a>: <a href="#BuildStep-URLs">BuildStep URLs</a></li> -<li><a href="#index-locks-90">locks</a>: <a href="#Interlocks">Interlocks</a></li> -<li><a href="#index-logfiles-4">logfiles</a>: <a href="#Logfiles">Logfiles</a></li> -<li><a href="#index-LogLineObserver-86">LogLineObserver</a>: <a href="#Adding-LogObservers">Adding LogObservers</a></li> -<li><a href="#index-LogObserver-85">LogObserver</a>: <a href="#Adding-LogObservers">Adding LogObservers</a></li> -<li><a href="#index-mail-113">mail</a>: <a href="#MailNotifier">MailNotifier</a></li> -<li><a href="#index-Mercurial-Checkout-59">Mercurial Checkout</a>: <a href="#Mercurial">Mercurial</a></li> -<li><a href="#index-PBListener-117">PBListener</a>: <a href="#PBListener">PBListener</a></li> -<li><a href="#index-Perforce-Update-67">Perforce Update</a>: <a href="#P4">P4</a></li> -<li><a href="#index-Philosophy-of-operation-2">Philosophy of operation</a>: <a href="#History-and-Philosophy">History and Philosophy</a></li> -<li><a href="#index-Properties-51">Properties</a>: <a href="#Using-Build-Properties">Using Build Properties</a></li> -<li><a href="#index-Properties-37">Properties</a>: <a href="#Defining-Global-Properties">Defining Global Properties</a></li> -<li><a href="#index-Properties-35">Properties</a>: <a href="#Buildslave-Specifiers">Buildslave Specifiers</a></li> -<li><a href="#index-Properties-20">Properties</a>: <a href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a></li> -<li><a href="#index-Properties-11">Properties</a>: <a href="#Build-Properties">Build Properties</a></li> -<li><a href="#index-Scheduler-6">Scheduler</a>: <a href="#Schedulers">Schedulers</a></li> -<li><a href="#index-statusgui-119">statusgui</a>: <a href="#statusgui">statusgui</a></li> -<li><a href="#index-SVN-Checkout-55">SVN Checkout</a>: <a href="#SVN">SVN</a></li> -<li><a href="#index-treeStableTimer-97">treeStableTimer</a>: <a href="#BuildFactory-Attributes">BuildFactory Attributes</a></li> -<li><a href="#index-Triggers-30">Triggers</a>: <a href="#Triggerable-Scheduler">Triggerable Scheduler</a></li> -<li><a href="#index-Users-10">Users</a>: <a href="#Users">Users</a></li> -<li><a href="#index-Version-Control-5">Version Control</a>: <a href="#Version-Control-Systems">Version Control Systems</a></li> -<li><a href="#index-Waterfall-110">Waterfall</a>: <a href="#HTML-Waterfall">HTML Waterfall</a></li> -<li><a href="#index-WebStatus-108">WebStatus</a>: <a href="#WebStatus">WebStatus</a></li> -<li><a href="#index-WithProperties-52">WithProperties</a>: <a href="#Using-Build-Properties">Using Build Properties</a></li> - </ul><div class="footnote"> -<hr> -<a name="texinfo-footnotes-in-document"></a><h4>Footnotes</h4><p class="footnote"><small>[<a name="fn-1" href="#fnd-1">1</a>]</small> this -@reboot syntax is understood by Vixie cron, which is the flavor -usually provided with linux systems. Other unices may have a cron that -doesn't understand @reboot</p> - - <p class="footnote"><small>[<a name="fn-2" href="#fnd-2">2</a>]</small> except Darcs, but -since the Buildbot never modifies its local source tree we can ignore -the fact that Darcs uses a less centralized model</p> - - <p class="footnote"><small>[<a name="fn-3" href="#fnd-3">3</a>]</small> many VC -systems provide more complexity than this: in particular the local -views that P4 and ClearCase can assemble out of various source -directories are more complex than we're prepared to take advantage of -here</p> - - <p class="footnote"><small>[<a name="fn-4" href="#fnd-4">4</a>]</small> Monotone's <em>multiple heads</em> feature -violates this assumption of cumulative Changes, but in most situations -the changes don't occur frequently enough for this to be a significant -problem</p> - - <p class="footnote"><small>[<a name="fn-5" href="#fnd-5">5</a>]</small> this <code>checkoutDelay</code> defaults -to half the tree-stable timer, but it can be overridden with an -argument to the Source Step</p> - - <p class="footnote"><small>[<a name="fn-6" href="#fnd-6">6</a>]</small> To be precise, it is an object or a list of objects -which all implement the <code>buildbot.interfaces.IChangeSource</code> -Interface. It is unusual to have multiple ChangeSources, so this key -accepts either a single ChangeSource or a sequence of them.</p> - - <p class="footnote"><small>[<a name="fn-7" href="#fnd-7">7</a>]</small> Build properties are serialized along with the -build results, so they must be serializable. For this reason, the -value of any build property should be simple inert data: strings, -numbers, lists, tuples, and dictionaries. They should not contain -class instances.</p> - - <p class="footnote"><small>[<a name="fn-8" href="#fnd-8">8</a>]</small> framboozle.com is still available. Remember, I get 10% -:).</p> - - <p class="footnote"><small>[<a name="fn-9" href="#fnd-9">9</a>]</small> Framboozle gets very excited about running unit -tests.</p> - - <p class="footnote"><small>[<a name="fn-10" href="#fnd-10">10</a>]</small> See -http://en.wikipedia.org/wiki/Read/write_lock_pattern for more information.</p> - - <p class="footnote"><small>[<a name="fn-11" href="#fnd-11">11</a>]</small> Deadlock is the situation where two or more slaves each -hold a lock in exclusive mode, and in addition want to claim the lock held by -the other slave exclusively as well. Since locks allow at most one exclusive -user, both slaves will wait forever.</p> - - <p class="footnote"><small>[<a name="fn-12" href="#fnd-12">12</a>]</small> Starving is the situation that only a few locks are available, -and they are immediately grabbed by another build. As a result, it may take a -long time before all locks needed by the starved build are free at the same -time.</p> - - <p class="footnote"><small>[<a name="fn-13" href="#fnd-13">13</a>]</small> It may even be possible to provide SSL access by using -a specification like -<code>"ssl:12345:privateKey=mykey.pen:certKey=cert.pem"</code>, but this is -completely untested</p> - - <p class="footnote"><small>[<a name="fn-14" href="#fnd-14">14</a>]</small> Apparently this is the same way -http://buildd.debian.org displays build status</p> - - <hr></div> - -</body></html> - |