diff options
Diffstat (limited to 'buildbot/NEWS')
| -rw-r--r-- | buildbot/NEWS | 2591 |
1 files changed, 0 insertions, 2591 deletions
diff --git a/buildbot/NEWS b/buildbot/NEWS deleted file mode 100644 index e12b3a8..0000000 --- a/buildbot/NEWS +++ /dev/null @@ -1,2591 +0,0 @@ -User visible changes in Buildbot. -*- outline -*- - -* Release 0.7.10p1 (2 Mar 2009) - -This is a bugfix release for 0.7.10, fixing a few minor bugs: - -** Bugs Fixed - -*** add a missing method to the IRC status plugin - -*** add RPM-related buildsteps to setup.py - -* Release 0.7.10 (25 Feb 2009) - -This release is mainly a collection of user-submitted patches since -the last release. - -** New Features - -*** Environment variables in a builder (#100) - -It is useful to be able to pass environment variables to all steps in a -builder. This is now possible by adding { .. 'env': { 'var' : 'value' }, ... } -to the builder specification. - -*** IRC status plugin improvements (#330, #357, #378, #280, #381, #411, #368) - -*** usePTY specified in master.cfg, defaults to False (#158, #255) - -Using a pty has some benefits in terms of supporting "Stop Build", but causes -numerous problems with simpler jobs which can be killed by a SIGHUP when their -standard input is closed. With this change, PTYs are not used by default, -although you can enable them either on slaves (with the --usepty option to -create-slave) or on the master. - -*** More information about buildslaves via the web plugin (#110) - -A new page, rooted at /buildslave/$SLAVENAME, gives extensive information about -the buildslave. - -*** More flexible merging of requests (#415) - -The optional c['mergeRequests'] configuration parameter takes a function -which can decide whether two requests are mergeable. - -*** Steps can be made to run even if the build has halted (#414) - -Adding alwaysRun=True to a step will cause it to run even if some other step -has failed and has haltOnFailure=True. - -*** Compress buildstep logfiles (#26) - -Logs for each buildstep, which can take a lot of space on a busy buildmaster, -are automatically compressed after the step has finished. - -*** Support for "latent" buildslaves - -The buildslaves that are started on-demand are called "latent" buildslaves. -Buildbot ships with an abstract base class for building latent buildslaves, and -a concrete implementation for AWS EC2. - -*** Customized MailNotifier messages (#175) - -MailNotifier now takes an optional function to build the notification message, -allowing ultimate site-level control over the format of buildbot's notification -emails. - -*** Nightly scheduler support for building only if changes have occurred - -With the addition of onlyIfChanged=True, the Nightly scheduler will not schedule -a new build if no changes have been made since its last scheduled build. - -*** Add ATOM/RSS feeds to WebStatus (#372) - -Two new pages, /atom and /rss, provide feeds of build events to any feed -reader. These paths take the same "category" and "branch" arguments as the -waterfall and grid. - -*** Add categories to Schedulers and Changes (#182) - -This allows a moderate amount of support for multiple projects built in a -single buildmaster. - -*** Gracefully shut down a buildslave after its build is complete - -The /buildslaves/$SLAVENAME pages have a "Gracefully Shutdown" button which -will cause the corresponding slave to shut itself down when it finishes its -current build. This is a good way to do work on a slave without causing a -spurious build failure. - -*** SVN source steps can send usernames and passwords (#41) - -Adding username="foo" and/or password="bar" to an SVN step will cause ---username and --password arguments to be passed to 'svn' on the slave side. -Passwords are suitably obfuscated in logfiles. - -** New Steps - -*** DirectoryUpload (#393) - -This step uploads an entire directory to the master, and can be useful when a -build creates several products (e.g., a client and server package). - -*** MasterShellCommand - -This step runs a shell command on the server, and can be useful for -post-processing build products, or performing other maintenance tasks on the -master. - -*** PyLint (#259) - -A PyLint step is available to complement the existing PyFlakes step. - -** Bugs Fixed - -*** Process output from new versions of Test::Harness (#346) - -*** Fixes to the try client and scheduler - -*** Remove redundant loop in MailNotifier (#315) - -*** Display correct $PWD in logfiles (#179) - -*** Do not assume a particular python version on Windows (#401) - -*** Sort files in changes (#402) - -*** Sort buildslaves lexically (#416) - -*** Send properties to all builds initiated by AnyBranchScheduler - -*** Dependent Schedulers are more robust to reconfiguration (#35) - -*** Fix properties handling in triggered buidls (#392) - -*** Use "call" on Windows to avoid errors (#417) - -*** Support setDefaultWorkdir in FileUpload and FileDownload (#209) - -*** Support WithProperties in FileUpload and FileDownload (#210) - -*** Fix a bug where changes could be lost on a master crash (#202) - -*** Remove color settings from non-presentation code (#251) - -*** Fix builders which stopped working after a PING (#349, #85) - -*** Isolate Python exceptions in status plugins (#388) - -*** Notify about slaves missing at master startup (#302) - -*** Fix tracebacks in web display after a reconfig (#176) - -** Version-Control Changes - -*** Many Mercurial fixes - - - Inrepo branch support finalized (source step + changegroup hook + test case) - (#65 #185 #187) - - - Reduced amount of full clones by separating clone with update into - clone/pull/update steps (#186, #227) (see #412 for future work here) - - - Fixed mercurial changegroup hook to work with Mercurial 1.1 API (#181, #380) - -*** Many git fixes - -*** Add got_revision to Perforce support (#127) - -*** Use "git foo" everywhere instead of deprecated "git-foo" - -** Minor Changes - -*** factory.addSteps (#317) - -If you have a common list of steps that are included in multiple factories, you -can use f.addSteps(steplist) to add them all at once. - -*** Twisted logfile rotation and cleanup (#108) - -By default, Buildbot now rotates and cleans up the (potentially voluminous) -twistd.log files. - -*** Prioritize build requests based on the time they wre submitted (#334) - -Balancing of load is a bit more fair, although not true load balancing. - -* Release 0.7.9 (15 Sep 2008) - -** New Features - -*** Configurable public_html directory (#162) - -The public_html/ directory, which provides static content for the WebStatus() -HTTP server, is now configurable. The default location is still the -public_html/ subdirectory of the buildmaster's base directory, but you can -change this by passing a suitable argument when creating the WebStatus() -instance in your master.cfg file: - - c['status'].append( WebStatus(8080, public_html="/var/www/buildbot") ) - -*** Lock access modes (#313) - -Albert Hofkamp added code to provide two distinct access modes to Locks: -"counting" and "exclusive". Locks can accept a configurable number of -"counting"-mode users, or a single "exclusive"-mode. For example, a Lock is -defined with maxCount=3, and then a 'compile' BuildStep uses this lock in -counting mode, while a 'cleanup' BuildStep uses this lock in exclusive mode. -Then, there can be one, two, or three simultaneous Builds in the compile step -(as long as there are no builds in the cleanup step). Only one build can be -in the cleanup step at a time, and if there is such a build in the cleanup -step, then the compile steps in other builds will wait for it to finish. -Please see the "Interlocks" section of the user's manual for more details. - -** Bugs Fixed - -*** Buildslave missing_timeout= fired too quickly (#211) - -By providing a missing_timeout= argument when creating the BuildSlave -instance, you can ask the buildmaster to send email if a buildslave is -disconnected for too long. A bug in the previous version caused this -notification to be sent too soon, rather than waiting until the timeout -period expired. This should be fixed now. - -*** Test command display fixed (#332) - -In the previous version, a steps.shell.Test step would display the parsed -test results (in the step's box on the waterfall display) in lieu of any -other descriptive text the step might provide. In this release, these two -pieces of information are combined. - -** Minor Changes - -The buildmaster's version is logged to its twistd.log file at startup. The -buildslave does the same, to its own logfile. - -Remote commands now record how long each command took. The "elapsedTime=" -message will appear in the step's main logfile. - -The "buildbot restart" command no longer fails if the buildbot wasn't already -running. - -The FileUpload and FileDownload steps now create their target directories -(and any missing intermediate directories) before writing to the destination -file. - -The per-build and per-step web pages now show the start, finish, and elapsed -time of their build or step. - -If a Subversion-based build is started with a mixture of Changes that specify -particular numeric revisions and "HEAD" Changes (which indicate that a trunk -checkout is desired), the build will use a trunk checkout. Previously this -would probably cause an error. It is not clear how this situation might -arise. - -** Compability With Other Tools - -The mercurial commit hook (buildbot.changes.hgbuildbot) in the previous -version doesn't work with hg-1.0 or later (it uses an API function that was -present in the hg-0.9.5 release, but was removed from hg-1.0). This -incompability has been fixed: the new version of buildbot should be -compatible with hg-1.0 and newer (and it probably retains compability with -hg-0.9.5 and earlier too). (#328) - -The Git tool has traditionally provided two ways to run each command, either -as subcommands of /usr/bin/git (like "git checkout"), or as individual tools -(like /usr/bin/git-checkout). The latter form is being removed in the -upcoming 1.6 Git release. Previous versions of Buildbot have used the -git-checkout form, and will break when Git is upgraded to 1.6 or beyond. The -new Buildbot release switches to the subcommand form. Note that this is a -change on the buildslave side. - -The Git checkout command will now use the default branch (as set in the -steps.source.Git() step definition) if the changes that it is building do not -specify some other branch to build. (#340) - -** Deprecation Schedule - -No features have been deprecated in this release, and no deprecated features -have been removed. As a reminder, the following deprecated features are -scheduled for removal in an upcoming release: - -c['sources'] (plural) was replaced by c['change_source'] (singular) in 0.7.6, -and will be removed by 0.8.0. - -c['bots'] was replaced by c['buildslaves'] in 0.7.6, and will be removed by -0.8.0 . c['bots'] only accepts BuildSlave instances, not name/passwd tuples. - -The html.Waterfall status target was replaced by html.WebStatus in 0.7.6, and -will be removed by 0.8.0. - - -* Release 0.7.8 (24 Jul 2008) - -** New features - -The IRC bot will respond to three new commands: 'notify' subscribes the -channel (or the sender, if the command is sent as a private "/msg") to hear -about build events. 'join' tells the bot to join some new IRC channel. -'leave' tells it to leave a channel. See the "IRC Bot" section of the User's -Manual for details. (#171) - -Build Steps now have "statistics", in addition to logfiles. These are used to -count things like how many tests passed or failed. There are methods to sum -these counters across all steps and display the results in the Build status. -The Waterfall display now shows the count of failed tests on the top-most box -in each column, using this mechanism. - -The new buildbot.steps.shell.PerlModuleTest step was added, to run Perl unit -tests. This is a wrapper around the regular ShellCommand that parses the -output of the standard perl unit test system and counts how many tests -passed/failed/etc. The results are put into the step's summary text, and a -count of tests passed/failed/skipped are tracked in the steps's statistics. -The factory.CPAN build factory has been updated to use this, so configuring a -Buildbot to test a perl module available from CPAN should be as easy as: - - s = source.CVS(cvsroot, cvsmodule) - f = factory.CPAN(s) - -Build Properties have been generalized: they remain associated with a single -Build, but the properties can be set from a variety of sources. In previous -releases, the Build itself would set properties like 'buildername', 'branch', -and 'revision' (the latter two indicating which version of the source code it -was trying to get), and the source-checkout BuildSteps would set a property -named 'got_revision' (to indicate what version of the soruce code it actually -got). In this release, the 'scheduler' property is set to indicate which -Scheduler caused the build to be started. In addition, the config file can -specify properties to be set on all Builds, or on all Builds for a specific -Builder. All these properties are available for interpolation into -ShellCommands and environment variables by using the WithProperties() marker. - -It may be easier to implement simple build parameterization (e.g. to upload -generated binaries to a specific directory, or to only perform long-running -tests on a nightly build instead of upon every checkin) by using these Build -Properties than to write custom BuildSteps. - -** Other improvements - -The /buildslaves web page shows which slaves are currently running builds. -Offline slaves are displayed in bold. - -Buildbot's setup.py now provides metadata to setuptools (if installed): an -entry_points script was added, and a dependency upon twisted-2.4.x or newer -was declared. This makes it more likely that 'easy_install buildbot' will -work. - -The MailNotifier class acquired a mode="passing" flag: in this mode, the -buildbot will only send mail about passing builds (versus only on failing -builds, or only on builds which failed when the previous build had passed). - -** Bugs fixed - -Don't display force/stop build buttons when build control is disabled (#246) - -When a build is waiting on a lock, don't claim that it has started (#107) - -Make SVN mode=copy tolerate symlinks on freebsd, "cp -rp" -> "cp -RPp" (#86) - -The svnpoller changesource now ignores branch deletion (#261) - -The Git unit tests should run even if the user has not told Git about their -username/email. - -The WebStatus /xmlrpc server's getStatus() method was renamed to the -more-accurate getLastBuildResults(). - -The TinderboxMailNotifier status output acquired an useChangeTime= argument. - -The bonsaipoller changesource got some fixes. - -** Deprecation Schedule - -No features have been deprecated in this release, and no deprecated features -have been removed. As a reminder, the following deprecated features are -scheduled for removal in an upcoming release: - -c['sources'] (plural) was replaced by c['change_source'] (singular) in 0.7.6, -and will be removed by 0.8.0. - -c['bots'] was replaced by c['buildslaves'] in 0.7.6, and will be removed by -0.8.0 . c['bots'] only accepts BuildSlave instances, not name/passwd tuples. - -The html.Waterfall status target was replaced by html.WebStatus in 0.7.6, and -will be removed by 0.8.0. - - - -* Release 0.7.7 (29 Mar 2008) - -** Things You Need To Know - -*** builder names must not start with an underscore (`_'). - -These are now reserved for internal buildbot purposes, such as the magic -"_all" pseudo-builder that the web pages use to allow force-build buttons -that start builds on all Builders at once. - -** New Features - -*** "buildbot checkconfig" - -The "buildbot checkconfig" command will look at your master.cfg file and tell -you if there are any problems with it. This can be used to test potential -changes to your config file before submitting them to the running -buildmaster. This is particularly useful to run just before doing "buildbot -restart", since the restart will fail if the config file has an error. By -running "buildbot checkconfig master.cfg && buildbot restart", you'll only -perform the restart if the config file was ok. Many thanks to Ben Hearsum for -the patch. - -*** Waterfall "?category=FOO" query-arguments - -The Waterfall page now accepts one or more "category=" query arguments in the -URL, to filter the display by categories. These behave a lot like the -"builder=" query argument. Thanks to Jermo Davann for the patch. - -** Bugs Fixed - -Many bugs were fixed, and many minor features were added. Many thanks to -Dustin Mitchell who fixed and coordinated many of these. Here is a terse -list, for more details, please see the Trac page for the 0.7.7 release, at -http://buildbot.net/trac/query?status=closed&milestone=0.7.7 : - -Many of the URLs generated by the buildbot were wrong. -Display of last-heard-from timestamps on the buildslaves web page were wrong. -Asking an IRC bot about a build waiting on a Lock should no longer crash. -Same for the web viewer. -Stop treating the encouraged info/ directory as leftover. -Add more force/stop build buttons. -Timestamps displayed on the waterfall now handle daylight savings properly. -p4poller no longer quits after a single failure. -Improved Git support, including 'try', branch, and revisions. -Buildslaves now use 'git', not 'cogito'. -Make older hg client/servers handle specific-revision builds properly. -Fix twisted.scripts._twistw problem on twisted-2.5.0 and windows. -Fix workdir= and env= on ShellCommands -Fix logfile-watching in 'buildbot start' on OS-X. -Fix ShellCommand crashes when the program emits >640kB of output per chunk. -New WarningCountingShellCommand step. -Fix TreeSize step. -Fix transfer.FileUpload/FileDownload crashes for large files. -Make 'buildbor reconfig' on windows tell you that it doesn't work. -Add a To: header to the mail sent by the slave-missing timeout. -Disable usePTY= for most unit tests, it makes some debian systems flunk tests. -Add 'absolute source stamps' -Add 'triggerable schedulers', and a buildstep to trigger them. -Remove buildbot.changes.freshcvsmail -Add new XMLRPC methods: getAllBuilders, getStatus, getLastBuilds. -Accept WithProperties in more places: env=, workdir=, others. -Use --no-auth-cache with SVN commands to avoid clobbering shared svn state. -Add hours/minutes/seconds in the waterfall's ETA display. -Trial: count Doctest lines too. -ShellCommand: record more info in the headers: stdin closing, PTY usage. -Make it possible to stop builds across reconfig boundaries. -SVN revision numbers are now passed as strings, which was breaking MailNotifier - -** Deprecation Schedule - -The changes.freshcvsmail change source was replaced by -changes.mail.FCMaildirSource in 0.7.6, and has been removed in 0.7.7 . - -c['sources'] (plural) was replaced by c['change_source'] (singular) in 0.7.6, -and will be removed by 0.8.0. - -c['bots'] was replaced by c['buildslaves'] in 0.7.6, and will be removed by -0.8.0 . c['bots'] only accepts BuildSlave instances, not name/passwd tuples. - -The html.Waterfall status target was replaced by html.WebStatus in 0.7.6, and -will be removed by 0.8.0. - - -* Release 0.7.6 (30 Sep 2007) - -** Things You Need To Know - -*** 'buildbot upgrade-master' - -Each time you install a new version of Buildbot, you should run the new -'buildbot upgrade-master' command on each of your pre-existing buildmasters. -This will add files and fix (or at least detect) incompatibilities between -your old config and the new code. - -*** new WebStatus page - -The Waterfall has been replaced by the more general WebStatus display, -described below. WebStatus serves static files from a new public_html/ -directory that lives in the buildmaster's basedir. Files like index.html, -buildbot.css, and robots.txt are served directly from that directory, so any -modifications you wish to make should be made to those files. In particular, -any custom CSS you've written should be copied into public_html/buildbot.css. -The 'upgrade-master' command will populate this directory for you. - -The old Waterfall page is deprecated, but it should continue to work for -another few releases. It is now a subclass of WebStatus which just replaces -the default root URL with another copy of the /waterfall resource. - -*** Compatibility: Python-2.3 or newer, Twisted-2.0 or newer - -No compatiblity losses here, buildbot-0.7.6 is compatible with the same -versions of python and twisted that 0.7.5 was. - -Buildbot is tested on a regular basis (http://buildbot.buildbot.net) against -nearly a full matrix of Python-(2.3,2.4,2.5) * Twisted-(2.0,2.1,2.2,2.4,2.5). - -*** New Buildbot Home Page - -Buildbot has moved to a new Trac instance at http://buildbot.net/ , and all -new bugs and tickets should be filed there. The old sourceforge bugs at -http://buildbot.sf.net/ will slowly be migrated over. Mailing lists are still -managed at sourceforge, and downloads are still available there. - -*** Changed/Deprecated master.cfg Keys and Classes - -c['sources'] (plural) has been replaced by c['change_source'] (singular). - -c['bots'] has been replaced by c['buildslaves'], and it expects a list of -BuildSlave instances instead of tuples. See below for more details. - -The 'freshcvsmail' change source has been deprecated, and will be removed in -the next release. - -The html.Waterfall status target has been deprecated, and replaced by -html.WebStatus . - -** New Features - -*** WebStatus - -The new WebStatus display is a superset of the old Waterfall. It contains a -waterfall as a sub-page, but it also contains pages with more compact -representations of recent build status. The "one_line_per_build" page -contains just that, and "one_box_per_builder" shows just the information from -the top of the waterfall page (last-finished-build and current-activity). - -The initial page (when you hit the root of the web site) is served from -index.html, and provides links to the Waterfall as well as the other pages. - -Most of these pages can be filtered by adding query arguments to the URL. -Adding "?builder=XYZ" will cause the page to only show results for the given -builder. Adding "?builder=XYZ&builder=ABC" will show results for either -builder. "?branch=trunk" will limit the results to builds that involved code -from the trunk. - -The /waterfall page has arguments to hide those annoying "buildslave -connected" messages, to start and and at arbitrary times, and to auto-refresh -at a chosen interval (with a hardcoded minimum of 15 seconds). It also has a -"help" page with forms that will help you add all of these nifty filtering -arguments. - -The recommended practice is to modify the index.html file to include links to -the filtered pages that you find most useful. - -Note that WebStatus defaults to allowForce=False, meaning that the display -will not offer or accept "Force Build" or "Stop Build" controls. (The old -Waterfall defaults to allowForce=True). - -The new WebStatus pages try very hard to use only relative links, making life -better when the Buildbot sits behind an HTTP reverse proxy. - -In addition, there is a rudimentary XMLRPC server run by the WebStatus -object. It only has two methods so far, but it will acquire more in the -future. The first customer of this is a project to add a buildbot plugin to -Trac. - -*** BuildFactory.addStep(Step(args)) - -BuildFactories can be set up either with a complete list of steps, or by -calling the .addStep() method repeatedly. The preferred way to provide a step -is by instantiating it, rather than giving a class/kwargs pair. This gives -the BuildStep class a chance to examine the arguments (and complain about -anything it doesn't like) while the config file is being read and problems -are being logged. For example, the old-style: - - from buildbot.process.factory import BuildFactory, s - steps = [s(CVS, cvsroot="blah", mode="copy"), - s(Compile, command=["make", "all"]), - s(Test, command=["make", "test"]), - ] - f = BuildFactory(steps) - -is now: - - f = BuildFactory() - f.addStep( CVS(cvsroot="blah", mode="copy") ) - f.addStep( Compile(command=["make", "all"]) ) - f.addStep( Test(command=["make", "test"]) ) - -Authors of BuildStep subclasses which override __init__ to add new arguments -must register them with self.addFactoryArguments(**newargs) to make sure that -those classes will work with this new style, otherwise the new arguments will -be lost. - -Using class/kwargs pairs is deprecated, and will be removed in a future -release. - - -*** BuildSlave instances, max_builds=, notify_on_missing= - -Buildslave specification has changed a lot in this release. The old config: - - c['bots'] = [ ("bot1name", "bot1passwd"), - ("bot2name", "bot2passwd") ] - -is now: - - from buildbot.buildslave import BuildSlave - c['slaves'] = [ BuildSlave("bot1name", "bot1passwd"), - BuildSlave("bot2name", "bot2passwd") ] - -This new form gives us the ability to add new controls. The first is -"max_builds=", which imposes a concurrency limit that is like the usual -SlaveLock, but gives the buildmaster the opportunity to find a different -slave to run the build. (the buildslave is chosen before the SlaveLock is -claimed, so pure SlaveLocks don't let you take full advantage of build -farms). - -The other addition is "notify_on_missing=", which accepts an email address -(or list of addresses), and sends a message when the buildslave has been -disconnected for more than an hour (configurable with missing_timeout=). This -may be useful when you expect that the buildslave hosts should be available -most of the time, and want to investigate the reasons that it went offline. - - -** Other Improvements - -The IRC bot has been refactored to make it easier to add instant-messaging -status delivery in the future. The IM plugins are not yet written, though. - -When multiple buildslaves are available for a given build, one of them will -be picked at random. In previous releases, the first one on the list was -always picked. This helps to add a certain measure of load-balancing. More -improvements will be made in the future. - -When the buildslave does a VC checkout step that requires clobbering the -build directory (i.e. in all modes except for 'update'), the buildslave will -first set the permissions on all build files to allow their deletion, before -it attempts to delete them. This should fix some problems in which a build -process left non-user-writable files lying around (frequently a result of -enthusiastic unit tests). - -The BuildStep's workdir= argument can now accept a WithProperties() -specification, allowing greater control over the workdir. - -Support for the 'Bazaar' version control system (/usr/bin/bzr) has been -added, using the buildbot.steps.source.Bzr class. This is a replacement for -the old 'Arch' (/usr/bin/tla and /usr/bin/baz) systems, which are still -supported by Buildbot with the source.Arch and source.Bazaar classes, -respectively. Unfortunately the old baz system claimed the 'Bazaar' classname -early, so the new system must use source.Bzr instead of the desired -source.Bazaar . A future release might change this. - -A rudimentary Gnome Panel applet is provided in contrib/bb_applet.py, which -provides 'buildbot statusgui' -like colored status boxes inside the panel. -Installing it is a bit tricky, though. - -The 'buildbot try' command now accepts a '--diff=foo.patch' argument, to let -you provide a pre-computed patch. This makes it easier to test out patches -that you've looked over for safety, without first applying them to your local -source tree. - -A new Mercurial change source was added, hg_buildbot.py, which runs as an -in-process post-commit hook. This gives us access to much more information -about the change, as well as being much faster. - -The email-based changesource have been refactored, to make it easier to write -new mail parsers. A parser for the SVN "commit-email.pl" script has been -added. - -** Bugs Fixed - -Far too many to count. Please see -http://buildbot.net/trac/query?status=closed&milestone=0.7.6 for a partial -list of tickets closed for this release, and the ChangeLog for a complete -list of all changes since 0.7.5 . - - -* Release 0.7.5 (10 Dec 2006) - -** Things You Need To Know - -*** The Great BuildStep Renaming - -All BuildSteps have moved! They used to be classes in buildbot.process.step, -but now they all have separate modules in buildbot.steps.* . They have been -split out into separate categories: for example, the source checkout steps -are now buildbot.steps.source.CVS, buildbot.steps.source.Darcs, etc. The most -commonly used one is probably buildbot.steps.shell.ShellCommand . The -python-specific steps are in buildbot.steps.python, and the Twisted-specific -steps are in buildbot.steps.python_twisted . - -You will need to update your master.cfg files to use the new names. The old -names are deprecated and will be removed altogether in the next release. - -*** Compatibility - -Buildbot now requires python-2.3 or later. Buildbot now requires -Twisted-2.0.0 or later. Support for earlier versions of both has finally been -removed. If you discover it works with unsupported versions, please return -your Buildbot to the factory for repairs :-). - -Buildbot has *not* yet been tested against the recent python-2.5 release. It -has been tested against the latest SVN version of Twisted, but only in -conjunction with python-2.4 . - -** new features - -*** reconfiguring a Builder no longer causes a disconnect/reconnect cycle - -This means that sending SIGHUP to the master or running 'buildbot reconfig -MASTERDIR' command no longer interrupts any current builds, nor does it lose -pending builds like it did before. This involved a fairly substantial -refactoring of the various internal BotPerspective/BotMaster/Builder classes. -Note that reconfiguring Schedulers still loses any Changes that were waiting -for the tree to become stable: hopefully this will be fixed in the next -release. - -*** 'buildbot start/restart/reconfig' now show logs until startup is complete - -These commands now have additional code to follow twistd.log and display all -the lines that are emitted from the beginning of the start/reconfig action -until it has completed. This gives you a chance to see any problems detected -in the config file without needing to manually look in twistd.log or use -another shell to 'tail -f' it. This also makes it clear which config file is -being used. This functionality is not available under windows. - -In addition, if any problems are detected during 'start' or 'restart' (but -not reconfig), the buildbot command will terminate with a non-zero exit -status, making it easier to use in scripts. Closes SF#1517975. - -*** Locks now take maxCount=N to allow multiple simultaneous owners - -This allows Locks to be non-exclusive but still limit maximum concurrency. -Thanks to James Knight for the patch. Closes SF#1434997. - -*** filetransfer steps - -buildbot.steps.transfer.FileUpload is a buildstep that will move files from -the slave to the master. Likewise, FileDownload will move files from the -master down to the buildslave. Many thanks to Albert Hofkamp for contributing -these classes. Closes SF#1504631. - -*** pyflakes step - -buildbot.steps.python.PyFlakes will run the simple 'pyflakes' static analysis -tool and parse the results to tell you about undefined names, unused imports, -etc. You'll need to tell it how to run pyflakes, usually with something like -command=["pyflakes", "src/packagedir"] or the like. The default command is -"make pyflakes", which assumes that you have a suitable target in your -top-level Makefile. - -*** Monotone support - -Nathaniel Smith has contributed initial support for the Monotone version -control system. The code still needs docs and tests, but on the other hand it -has been in use by the Monotone buildbot for a long time now, so it is -probably fairly stable. - -*** Tinderbox support - -Ben Hearsum and the Mozilla crew have contributed some classes to allow -Buildbot to work with Tinderbox clients. One piece is -buildbot.changes.bonsaipoller.BonsaiPoller, which is a ChangeSource that -polls a Bonsai server (which is a kind of web-vased viewcvs CGI script) to -discover source code changes. The other piece is -buildbot.status.tinderbox.TinderboxMailNotifier, which is a status plugin -that sends email in the same format as Tinderbox does, which allows a number -of Tinderbox tools to be driven by Buildbot instead. - -*** SVN Poller - -Niklaus Giger contributed a ChangeSource (buildbot.changes.svnpoller) which -polls a remote SVN repository on a periodic basis. This is useful when, for -whatever reason, you cannot add a post-commit hook script to the repository. -This obsoletes the external contrib/svn_watcher.py script. - -** notes for plugin developers - -*** IStatusLog.readlines() - -This new method makes it easier for a status plugin (or a -BuildStep.createSummary method) to walk through a StatusLog one line at a -time. For example, if you wanted to create an extra logfile that just -contained all the GCC warnings from the main log, you could use the -following: - - def createSummary(self, log): - warnings = [] - for line in log.readlines(): - if "warning:" in line: - warnings.append() - self.addCompleteLog('warnings', "".join(warnings)) - -The "BuildStep LogFiles" section of the user's manual contains more -information. This method is not particularly memory-efficient yet (it reads -the whole logfile into memory first, then splits it into lines); this will be -improved in a future release. - -** bug fixes - -*** Update source.SVN to work with the new SVN-1.4.0 - -The latest subversion changed the behavior in an unusual situation which -caused the unit tests to fail. This was unlikely to cause a problem in actual -usage, but the tests have been updated to pass with the new version. - -*** update svn_buildbot.py to avoid mangling filenames - -Older versions of this script were stripping the wrong number of columns from -the output of 'svnlook changed', and would sometimes mangle filenames. This -has been fixed. Closes SF#1545146. - -*** logfiles= caused subsequent build failures under Windows - -Earlier versions of buildbot didn't explicitly close any logfiles= file -handles when the build finished. On windows (where you cannot delete a file -that someone else is reading), this could cause the next build to fail as the -source checkout step was unable to delete the old working directory. This has -been fixed. Closes SF#1568415. - -*** logfiles= didn't work on OS-X - -Macintosh OS-X has a different behavior when reading files that have reached -EOF, the result was that logfiles= sometimes didn't work. Thanks to Mark Rowe -for the patch. - -** other changes - -The 'buildbot sighup MASTERDIR' command has been replaced with 'buildbot -reconfig MASTERDIR', since that seems to be a slightly more meaningful name. -The 'sighup' form will remain as an alias. - - -* Release 0.7.4 (23 Aug 2006) - -** Things You Need To Know - -The PBChangeSource's prefix= argument has changed, you probably need to add a -slash now. This is mostly used by sites which use Subversion and -svn_buildbot.py. - -The subcommands that are used to create a buildmaster or a buildslave have -changed. They used to be called 'buildbot master' and 'buildbot slave'. Now -they are called 'buildbot create-master' and 'buildbot create-slave'. Zipf's -Law suggests that these are more appropriate names for these -infrequently-used commands. - -The syntax for the c['manhole'] feature has changed. - -** new features - -*** full Perforce support - -SF#1473939: large patch from Scott Lamb, with docs and unit tests! This -includes both the step.P4 source-checkout BuildStep, and the changes.p4poller -ChangeSource you'll want to feed it. P4 is now supported just as well as all -the other VC systems. Thanks Scott! - -*** SSH-based Manhole - -The 'manhole' feature allows buildbot developers to get access to a python -read/eval/print loop (REPL) inside the buildmaster through a network -connection. Previously, this ran over unencrypted telnet, using a simple -username/password for access control. The new release defaults to encrypted -SSH access, using either username/password or an authorized_keys file (just -like sshd). There also exists an unencrypted telnet form, but its use is -discouraged. The syntax for setting up a manhole has changed, so master.cfg -files that use them must be updated. The "Debug options" section in the -user's manual provides a complete description. - -*** Multiple Logfiles - -BuildSteps can watch multiple log files in realtime, not just stdout/stderr. -This works in a similar fashion to 'tail -f': the file is polled once per -second, and any new data is sent to the buildmaster. - -This requires a buildslave running 0.7.4 or later, and a warning message is -produced if used against an old buildslave (which will otherwise produce no -data). Use "logfiles={'name': 'filename'}" to take advantage of this feature -from master.cfg, and see the "ShellCommand" section of the user's manual for -full documentation. - -The 'Trial' buildstep has been updated to use this, to display -_trial_temp/test.log in realtime. It also knows to fall back to the previous -"cat" command if the buildslave is too old. - -*** BuildStep URLs - -BuildSteps can now add arbitrary URLs which will be displayed on the -Waterfall page in the same place that Logs are presented. This is intended to -provide a link to generated HTML pages, such as the output of a code coverage -tool. The step is responsible for somehow uploading the HTML to a web server: -this feature merely provides an easy way to present the HREF link to the -user. See the "BuildStep URLs" section of the user's manual for details and -examples. - -*** LogObservers - -BuildSteps can now attach LogObservers to various logfiles, allowing them to -get real-time log output. They can use this to watch for progress-indicating -events (like counting the number of files compiled, or the number of tests -which have run), and update both ETA/progress-tracking and step text. This -allows for more accurate ETA information, and more information passed to the -user about how much of the process has completed. - -The 'Trial' buildstep has been updated to use this for progress tracking, by -counting how many test cases have run. - -** new documentation - -What classes are useful in your master.cfg file? A table of them has been -added to the user's manual, in a section called "Index of Useful Classes". - -Want a list of all the keys in master.cfg? Look in the "Index of master.cfg -keys" section. - -A number of pretty diagrams have been added to the "System Architecture" -portion of the manual, explaining how all the buildbot pieces fit together. - -An HTML form of the user's manual is now shipped in the source tarball. This -makes it a bit bigger: sorry about that. The old PyCon-2003 paper has been -removed from the distribution, as it is mostly supplanted by the user's -manual by this point. - -** bugfixes - -SF#1217699 + SF#1381867: The prefix= argument to PBChangeSource has been -changed: now it does just a simple string-prefix match and strip. The -previous behavior was buggy and unhelpful. NOTE: if you were using prefix= -before, you probably need to add a slash to the end of it. - -SF#1398174: ignore SVN property changes better, fixed by Olivier Bonnet - -SF#1452801: don't double-escape the build URL, fixed by Olivier Bonnet - -SF#1401121: add support for running py2exe on windows, by Mark Hammond - -reloading unchanged config files with WithProperties shouldn't change anything. - -All svn commands now include --non-interactive so they won't ask for -passwords. Instead, the command will fail if it cannot be performed without -user input. - -Deprecation warnings with newer versions of Twisted have been hushed. - -** compatibility - -I haven't actually removed support for Twisted-1.3.0 yet, but I'd like to. - -The step_twisted default value for --reporter matches modern Twisteds, -though, and won't work under 1.3.0. - -ShellCommand.flunkOnFailure now defaults to True, so any shell command which -fails counts as a build failure. Set this to False if you don't want this -behavior. - -** minor features - -contrib/darcs_buildbot.py contains a new script suitable for use in a darcs -commit-hook. - -Hovering a cursor over the yellow "Build #123" box in the Waterfall display -will pop up an HTML tooltip to show the reason for the build. Thanks to Zandr -Milewski for the suggestion. - -contrib/CSS/*.css now contains several contributed stylesheets to make the -Waterfall display a bit less ugly. Thanks to John O'Duinn for gathering them. - -ShellCommand and its derivatives can now accept either a string or a list of -strings in the description= and descriptionDone= arguments. Thanks to Paul -Winkler for the catch. - - -* Release 0.7.3 (23 May 2006) - -** compatibility - -This release is compatible with Twisted-1.3.0, but the next one will not be. -Please upgrade to at least Twisted-2.0.x soon, as the next buildbot release -will require it. - -** new features - -*** Mercurial support - -Support for Mercurial version control system (http://selenic.com/mercurial) -has been added. This adds a buildbot.process.step.Mercurial BuildStep. A -suitable hook script to deliver changes to the buildmaster is still missing. - -*** 'buildbot restart' command - -The 'buildbot restart BASEDIR' command will perform a 'buildbot stop' and -'buildbot start', and will attempt to wait for the buildbot process to shut -down in between. This is useful when you need to upgrade the code on your -buildmaster or buildslave and want to take it down for a minimum amount of -time. - -*** build properties - -Each build now has a set of named "Build Properties", which can be set by -steps and interpolated into ShellCommands. The 'revision' and 'got_revision' -properties are the most interesting ones available at this point, and can be -used e.g. to get the VC revision number into the filename of a generated -tarball. See the user's manual section entited "Build Properties" for more -details. - -** minor features - -*** IRC now takes password= argument - -Useful for letting your bot claim a persistent identity. - -*** svn_buildbot.py is easier to modify to understand branches -*** BuildFactory has a new .addStep method -*** p4poller has new arguments -*** new contrib scripts: viewcvspoll, svnpoller, svn_watcher - -These poll an external VC repository to watch for changes, as opposed to -adding a hook script to the repository that pushes changes into the -buildmaster. This means higher latency but may be easier to configure, -especially if you do not have authority on the repository host. - -*** VC build property 'got_revision' - -The 'got_revision' property reports what revision a VC step actually -acquired, which may be useful to know when building from HEAD. - -*** improved CSS in Waterfall - -The Waterfall display has a few new class= tags, which may make it easier to -write custom CSS to make it look prettier. - -*** robots_txt= argument in Waterfall - -You can now pass a filename to the robots_txt= argument, which will be served -as the "robots.txt" file. This can be used to discourage search engine -spiders from crawling through the numerous build-status pages. - -** bugfixes - -*** tests more likely to pass on non-English systems - -The unit test suite now sets $LANG='C' to make subcommands emit error -messages in english instead of whatever native language is in use on the -host. This improves the chances that the unit tests will pass on such -systems. This affects certain VC-related subcommands too. - -test_vc was assuming that the system time was expressed with a numeric -timezone, which is not always the case, especially under windows. This -probably works better now than it did before. This only affects the CVS -tests. - -'buildbot try' (for CVS) now uses UTC instead of the local timezone. The -'got_revision' property is also expressed in UTC. Both should help deal with -buggy versions of CVS that don't parse numeric timezones properly. - - -* Release 0.7.2 (17 Feb 2006) - -** new features - -*** all TCP port numbers in config file now accept a strports string - -Sometimes it is useful to restrict certain TCP ports that the buildmaster -listens on to use specific network interfaces. In particular, if the -buildmaster and SVN repository live on the same machine, you may want to -restrict the PBChangeSource to only listen on the loopback interface, -insuring that no external entities can inject Changes into the buildbot. -Likewise, if you are using something like Apache's reverse-proxy feature to -provide access to the buildmaster's HTML status page, you might want to hide -the real Waterfall port by having it only bind to the loopback interface. - -To accomplish this, use a string like "tcp:12345:interface=127.0.0.1" instead -of a number like 12345. These strings are called "strports specification -strings", and are documented in twisted's twisted.application.strports module -(you can probably type 'pydoc twisted.application.strports' to see this -documentation). Pretty much everywhere the buildbot takes a port number will -now accept a strports spec, and any bare numbers are translated into TCP port -numbers (listening on all network interfaces) for compatibility. - -*** buildslave --umask control - -Twisted's daemonization utility (/usr/bin/twistd) automatically sets the -umask to 077, which means that all files generated by both the buildmaster -and the buildslave will only be readable by the account under which the -respective daemon is running. This makes it unnecessarily difficult to share -build products (e.g. by symlinking ~/public_html/current_docs/ to a directory -within the slave's build directory where each build puts the results of a -"make docs" step). - -The 'buildbot slave <PARAMS>' command now accepts a --umask argument, which -can be used to override the umask set by twistd. If you create the buildslave -with '--umask=022', then all build products will be world-readable, making it -easier for other processes (run under other accounts) to access them. - -** bug fixes - -The 0.7.1 release had a bug whereby reloading the config file could break all -configured Schedulers, causing them to raise an exception when new changes -arrived but not actually schedule a new build. This has been fixed. - -Fixed a bug which caused the AnyBranchScheduler to explode when branch==None. -Thanks to Kevin Turner for the catch. I also think I fixed a bug whereby the -TryScheduler would explode when it was given a Change (which it is supposed -to simply ignore). - -The Waterfall display now does more quoting of names (including Builder -names, BuildStep names, etc), so it is more likely that these names can -contain unusual characters like spaces, quotes, and slashes. There may still -be some problems with these kinds of names, however.. please report any bugs -to the mailing list. - - -* Release 0.7.1 (26 Nov 2005) - -** new features - -*** scheduler.Nightly - -Dobes Vandermeer contributed a cron-style 'Nightly' scheduler. Unlike the -more-primitive Periodic class (which only lets you specify the duration -between build attempts), Nightly lets you schedule builds for specific times -of day, week, month, or year. The interface is very much like the crontab(5) -file. See the buildbot.scheduler.Nightly docstring for complete details. - -** minor new features - -*** step.Trial can work with Trial from Twisted >2.1.0 - -The 'Trial' step now accepts the trialMode= argument, which should be a list -of strings to be added to trial's argv array. This defaults to ["-to"], which -is appropriate for the Trial that ships in Twisted-2.1.0 and earlier, and -tells Trial to emit non-colorized verbose output. To use this step with -trials from later versions of Twisted, this should be changed to -["--reporter=bwverbose"]. - -In addition, you can now set other Trial command-line parameters through the -trialArgs= argument. This is a list of strings, and defaults to an empty list. - -*** Added a 'resubmit this build' button to the web page - -*** Make the VC-checkout step's description more useful - -Added the word "[branch]" to the VC step's description (used in the Step's -box on the Waterfall page, among others) when we're checking out a -non-default branch. Also add "rNNN" where appropriate to indicate which -revision is being checked out. Thanks to Brad Hards and Nathaniel Smith for -the suggestion. - -** bugs fixed - -Several patches from Dobes Vandermeer: Escape the URLs in email, in case they -have spaces and such. Fill otherwise-empty <td> elements, as a workaround for -buggy browsers that might optimize them away. Also use binary mode when -opening status pickle files, to make windows work better. The -AnyBranchScheduler now works even when you don't provide a fileIsImportant= -argument. - -Stringify the base revision before stuffing it into a 'try' jobfile, helping -SVN and Arch implement 'try' builds better. Thanks to Steven Walter for the -patch. - -Fix the compare_attrs list in PBChangeSource, FreshCVSSource, and Waterfall. -Before this, certain changes to these objects in the master.cfg file were -ignored, such that you would have to stop and re-start the buildmaster to -make them take effect. - -The config file is now loaded serially, shutting down old (or replaced) -Status/ChangeSource plugins before starting new ones. This fixes a bug in -which changing an aspect of, say, the Waterfall display would cause an -exception as both old and new instances fight over the same TCP port. This -should also fix a bug whereby new Periodic Schedulers could fire a build -before the Builders have finished being added. - -There was a bug in the way Locks were handled when the config file was -reloaded: changing one Builder (but not the others) and reloading master.cfg -would result in multiple instances of the same Lock object, so the Locks -would fail to prevent simultaneous execution of Builds or Steps. This has -been fixed. - -** other changes - -For a long time, certain StatusReceiver methods (like buildStarted and -stepStarted) have been able to return another StatusReceiver instance -(usually 'self') to indicate that they wish to subscribe to events within the -new object. For example, if the buildStarted() method returns 'self', the -status receiver will also receive events for the new build, like -stepStarted() and buildETAUpdate(). Returning a 'self' from buildStarted() is -equivalent to calling build.subscribe(self). - -Starting with buildbot-0.7.1, this auto-subscribe convenience will also -register to automatically unsubscribe the target when the build or step has -finished, just as if build.unsubscribe(self) had been called. Also, the -unsubscribe() method has been changed to not explode if the same receiver is -unsubscribed multiple times. (note that it will still explode is the same -receiver is *subscribed* multiple times, so please continue to refrain from -doing that). - - -* Release 0.7.0 (24 Oct 2005) - -** new features - -*** new c['schedulers'] config-file element (REQUIRED) - -The code which decides exactly *when* a build is performed has been massively -refactored, enabling much more flexible build scheduling. YOU MUST UPDATE -your master.cfg files to match: in general this will merely require you to -add an appropriate c['schedulers'] entry. Any old ".treeStableTime" settings -on the BuildFactory instances will now be ignored. The user's manual has -complete details with examples of how the new Scheduler classes work. - -*** c['interlocks'] removed, Locks and Dependencies now separate items - -The c['interlocks'] config element has been removed, and its functionality -replaced with two separate objects. Locks are used to tell the buildmaster -that certain Steps or Builds should not run at the same time as other Steps -or Builds (useful for test suites that require exclusive access to some -external resource: of course the real fix is to fix the tests, because -otherwise your developers will be suffering from the same limitations). The -Lock object is created in the config file and then referenced by a Step -specification tuple or by the 'locks' key of the Builder specification -dictionary. Locks come in two flavors: MasterLocks are buildmaster-wide, -while SlaveLocks are specific to a single buildslave. - -When you want to have one Build run or not run depending upon whether some -other set of Builds have passed or failed, you use a special kind of -Scheduler defined in the scheduler.Dependent class. This scheduler watches an -upstream Scheduler for builds of a given source version to complete, and only -fires off its own Builders when all of the upstream's Builders have built -that version successfully. - -Both features are fully documented in the user's manual. - -*** 'buildbot try' - -The 'try' feature has finally been added. There is some configuration -involved, both in the buildmaster config and on the developer's side, but -once in place this allows the developer to type 'buildbot try' in their -locally-modified tree and to be given a report of what would happen if their -changes were to be committed. This works by computing a (base revision, -patch) tuple that describes the developer's tree, sending that to the -buildmaster, then running a build with that source on a given set of -Builders. The 'buildbot try' tool then emits status messages until the builds -have finished. - -'try' exists to allow developers to run cross-platform tests on their code -before committing it, reducing the chances they will inconvenience other -developers by breaking the build. The UI is still clunky, but expect it to -change and improve over the next few releases. - -Instructions for developers who want to use 'try' (and the configuration -changes necessary to enable its use) are in the user's manual. - -*** Build-On-Branch - -When suitably configured, the buildbot can be used to build trees from a -variety of related branches. You can set up Schedulers to build a tree using -whichever branch was last changed, or users can request builds of specific -branches through IRC, the web page, or (eventually) the CLI 'buildbot force' -subcommand. - -The IRC 'force' command now takes --branch and --revision arguments (not that -they always make sense). Likewise the HTML 'force build' button now has an -input field for branch and revision. Your build's source-checkout step must -be suitably configured to support this: for SVN it involves giving both a -base URL and a default branch. Other VC systems are configured differently. -The ChangeSource must also provide branch information: the 'buildbot -sendchange' command now takes a --branch argument to help hook script writers -accomplish this. - -*** Multiple slaves per Builder - -You can now attach multiple buildslaves to each Builder. This can provide -redundancy or primitive load-balancing among many machines equally capable of -running the build. To use this, define a key in the Builder specification -dictionary named 'slavenames' with a list of buildslave names (instead of the -usual 'slavename' that contains just a single slavename). - -*** minor new features - -The IRC and email status-reporting facilities now provide more specific URLs -for particular builds, in addition to the generic buildmaster home page. The -HTML per-build page now has more information. - -The Twisted-specific test classes have been modified to match the argument -syntax preferred by Trial as of Twisted-2.1.0 and newer. The generic trial -steps are still suitable for the Trial that comes with older versions of -Twisted, but may produce deprecation warnings or errors when used with the -latest Trial. - -** bugs fixed - -DNotify, used by the maildir-watching ChangeSources, had problems on some -64-bit systems relating to signed-vs-unsigned constants and the DN_MULTISHOT -flag. A workaround was provided by Brad Hards. - -The web status page should now be valid XHTML, thanks to a patch by Brad -Hards. The charset parameter is specified to be UTF-8, so VC comments, -builder names, etc, should probably all be in UTF-8 to be displayed properly. - -** creeping version dependencies - -The IRC 'force build' command now requires python2.3 (for the shlex.split -function). - - -* Release 0.6.6 (23 May 2005) - -** bugs fixed - -The 'sendchange', 'stop', and 'sighup' subcommands were broken, simple bugs -that were not caught by the test suite. Sorry. - -The 'buildbot master' command now uses "raw" strings to create .tac files -that will still function under windows (since we must put directory names -that contain backslashes into that file). - -The keep-on-disk behavior added in 0.6.5 included the ability to upgrade old -in-pickle LogFile instances. This upgrade function was not added to the -HTMLLogFile class, so an exception would be raised when attempting to load or -display any build with one of these logs (which are normally used only for -showing build exceptions). This has been fixed. - -Several unnecessary imports were removed, so the Buildbot should function -normally with just Twisted-2.0.0's "Core" module installed. (of course you -will need TwistedWeb, TwistedWords, and/or TwistedMail if you use status -targets that require them). The test suite should skip all tests that cannot -be run because of missing Twisted modules. - -The master/slave's basedir is now prepended to sys.path before starting the -daemon. This used to happen implicitly (as a result of twistd's setup -preamble), but 0.6.5 internalized the invocation of twistd and did not copy -this behavior. This change restores the ability to access "private.py"-style -modules in the basedir from the master.cfg file with a simple "import -private" statement. Thanks to Thomas Vander Stichele for the catch. - - -* Release 0.6.5 (18 May 2005) - -** deprecated config keys removed - -The 'webPortnum', 'webPathname', 'irc', and 'manholePort' config-file keys, -which were deprecated in the previous release, have now been removed. In -addition, Builders must now always be configured with dictionaries: the -support for configuring them with tuples has been removed. - -** master/slave creation and startup changed - -The buildbot no longer uses .tap files to store serialized representations of -the buildmaster/buildslave applications. Instead, this release now uses .tac -files, which are human-readable scripts that create new instances (rather -than .tap files, which were pickles of pre-created instances). 'mktap -buildbot' is gone. - -You will need to update your buildbot directories to handle this. The -procedure is the same as creating a new buildmaster or buildslave: use -'buildbot master BASEDIR' or 'buildbot slave BASEDIR ARGS..'. This will -create a 'buildbot.tac' file in the target directory. The 'buildbot start -BASEDIR' will use twistd to start the application. - -The 'buildbot start' command now looks for a Makefile.buildbot, and if it -finds one (and /usr/bin/make exists), it will use it to start the application -instead of calling twistd directly. This allows you to customize startup, -perhaps by adding environment variables. The setup commands create a sample -file in Makefile.sample, but you must copy this to Makefile.buildbot to -actually use it. The previous release looked for a bare 'Makefile', and also -installed a 'Makefile', so you were always using the customized approach, -even if you didn't ask for it. That old Makefile launched the .tap file, so -changing names was also necessary to make sure that the new 'buildbot start' -doesn't try to run the old .tap file. - -'buildbot stop' now uses os.kill instead of spawning an external process, -making it more likely to work under windows. It waits up to 5 seconds for the -daemon to go away, so you can now do 'buildbot stop BASEDIR; buildbot start -BASEDIR' with less risk of launching the new daemon before the old one has -fully shut down. Likewise, 'buildbot start' imports twistd's internals -directly instead of spawning an external copy, so it should work better under -windows. - -** new documentation - -All of the old Lore-based documents were converted into a new Texinfo-format -manual, and considerable new text was added to describe the installation -process. The docs are not yet complete, but they're slowly shaping up to form -a proper user's manual. - -** new features - -Arch checkouts can now use precise revision stamps instead of always using -the latest revision. A separate Source step for using Bazaar (an alternative -Arch client) instead of 'tla' was added. A Source step for Cogito (the new -linux kernel VC system) was contributed by Brandon Philips. All Source steps -now accept a retry= argument to indicate that failing VC checkouts should be -retried a few times (SF#1200395), note that this requires an updated -buildslave. - -The 'buildbot sendchange' command was added, to be used in VC hook scripts to -send changes at a pb.PBChangeSource . contrib/arch_buildbot.py was added to -use this tool; it should be installed using the 'Arch meta hook' scheme. - -Changes can now accept a branch= parameter, and Builders have an -isBranchImportant() test that acts like isFileImportant(). Thanks to Thomas -Vander Stichele. Note: I renamed his tag= to branch=, in anticipation of an -upcoming feature to build specific branches. "tag" seemed too CVS-centric. - -LogFiles have been rewritten to stream the incoming data directly to disk -rather than keeping a copy in memory all the time (SF#1200392). This -drastically reduces the buildmaster's memory requirements and makes 100MB+ -log files feasible. The log files are stored next to the serialized Builds, -in files like BASEDIR/builder-dir/12-log-compile-output, so you'll want a -cron job to delete old ones just like you do with old Builds. Old-style -Builds from 0.6.4 and earlier are converted when they are first read, so the -first load of the Waterfall display after updating to this release may take -quite some time. - -** build process updates - -BuildSteps can now return a status of EXCEPTION, which terminates the build -right away. This allows exceptions to be caught right away, but still make -sure the build stops quickly. - -** bug fixes - -Some more windows incompatibilities were fixed. The test suite now has two -failing tests remaining, both of which appear to be Twisted issues that -should not affect normal operation. - -The test suite no longer raises any deprecation warnings when run against -twisted-2.0 (except for the ones which come from Twisted itself). - - -* Release 0.6.4 (28 Apr 2005) - -** major bugs fixed - -The 'buildbot' tool in 0.6.3, when used to create a new buildmaster, failed -unless it found a 'changes.pck' file. As this file is created by a running -buildmaster, this made 0.6.3 completely unusable for first-time -installations. This has been fixed. - -** minor bugs fixed - -The IRC bot had a bug wherein asking it to watch a certain builder (the "I'll -give a shout when the build finishes" message) would cause an exception, so -it would not, in fact, shout. The HTML page had an exception in the "change -sources" page (reached by following the "Changes" link at the top of the -column that shows the names of commiters). Re-loading the config file while -builders were already attached would result in a benign error message. The -server side of the PBListener status client had an exception when providing -information about a non-existent Build (e.g., when the client asks for the -Build that is currently running, and the server says "None"). - -These bugs have all been fixed. - -The unit tests now pass under python2.2; they were failing before because of -some 2.3isms that crept in. More unit tests which failed under windows now -pass, only one (test_webPathname_port) is still failing. - -** 'buildbot' tool looks for a .buildbot/options file - -The 'statusgui' and the 'debugclient' subcommands can both look for a -.buildbot/ directory, and an 'options' file therein, to extract default -values for the location of the buildmaster. This directory is searched in the -current directory, its parent, etc, all the way up to the filesystem root -(assuming you own the directories in question). It also look in ~/.buildbot/ -for this file. This feature allows you to put a .buildbot at the top of your -working tree, telling any 'buildbot' invocations you perform therein how to -get to the buildmaster associated with that tree's project. - -Windows users get something similar, using %APPDATA%/buildbot instead of -~/.buildbot . - -** windows ShellCommands are launched with 'cmd.exe' - -The buildslave has been modified to run all list-based ShellCommands by -prepending [os.environ['COMSPEC'], '/c'] to the argv list before execution. -This should allow the buildslave's PATH to be searched for commands, -improving the chances that it can run the same 'trial -o foo' commands as a -unix buildslave. The potential downside is that spaces in argv elements might -be re-parsed, or quotes might be re-interpreted. The consensus on the mailing -list was that this is a useful thing to do, but please report any problems -you encounter with it. - -** minor features - -The Waterfall display now shows the buildbot's home timezone at the top of -the timestamp column. The default favicon.ico is now much nicer-looking (it -is generated with Blender.. the icon.blend file is available in CVS in -docs/images/ should you care to play with it). - - - -* Release 0.6.3 (25 Apr 2005) - -** 'buildbot' tool gets more uses - -The 'buildbot' executable has acquired three new subcommands. 'buildbot -debugclient' brings up the small remote-control panel that connects to a -buildmaster (via the slave port and the c['debugPassword']). This tool, -formerly in contrib/debugclient.py, lets you reload the config file, force -builds, and simulate inbound commit messages. It requires gtk2, glade, and -the python bindings for both to be installed. - -'buildbot statusgui' brings up a live status client, formerly available by -running buildbot/clients/gtkPanes.py as a program. This connects to the PB -status port that you create with: - - c['status'].append(client.PBListener(portnum)) - -and shows two boxes per Builder, one for the last build, one for current -activity. These boxes are updated in realtime. The effect is primitive, but -is intended as an example of what's possible with the PB status interface. - -'buildbot statuslog' provides a text-based running log of buildmaster events. - -Note: command names are subject to change. These should get much more useful -over time. - -** web page has a favicon - -When constructing the html.Waterfall instance, you can provide the filename -of an image that will be provided when the "favicon.ico" resource is -requested. Many web browsers display this as an icon next to the URL or -bookmark. A goofy little default icon is included. - -** web page has CSS - -Thanks to Thomas Vander Stichele, the Waterfall page is now themable through -CSS. The default CSS is located in buildbot/status/classic.css, and creates a -page that is mostly identical to the old, non-CSS based table. - -You can specify a different CSS file to use by passing it as the css= -argument to html.Waterfall(). See the docstring for Waterfall for some more -details. - -** builder "categories" - -Thomas has added code which places each Builder in an optional "category". -The various status targets (Waterfall, IRC, MailNotifier) can accept a list -of categories, and they will ignore any activity in builders outside this -list. This makes it easy to create some Builders which are "experimental" or -otherwise not yet ready for the world to see, or indicate that certain -builders should not harass developers when their tests fail, perhaps because -the build slaves for them are not yet fully functional. - -** Deprecated features - -*** defining Builders with tuples is deprecated - -For a long time, the preferred way to define builders in the config file has -been with a dictionary. The less-flexible old style of a 4-item tuple (name, -slavename, builddir, factory) is now officially deprecated (i.e., it will -emit a warning if you use it), and will be removed in the next release. -Dictionaries are more flexible: additional keys like periodicBuildTime are -simply unavailable to tuple-defined builders. - -Note: it is a good idea to watch the logfile (usually in twistd.log) when you -first start the buildmaster, or whenever you reload the config file. Any -warnings or errors in the config file will be found there. - -*** c['webPortnum'], c['webPathname'], c['irc'] are deprecated - -All status reporters should be defined in the c['status'] array, using -buildbot.status.html.Waterfall or buildbot.status.words.IRC . These have been -deprecated for a while, but this is fair warning that these keys will be -removed in the next release. - -*** c['manholePort'] is deprecated - -Again, this has been deprecated for a while, in favor of: - - c['manhole'] = master.Manhole(port, username, password) - -The preferred syntax will eventually let us use other, better kinds of debug -shells, such as the experimental curses-based ones in the Twisted sandbox -(which would offer command-line editing and history). - -** bug fixes - -The waterfall page has been improved a bit. A circular-reference bug in the -web page's TextLog class was fixed, which caused a major memory leak in a -long-running buildmaster with large logfiles that are viewed frequently. -Modifying the config file in a way which only changed a builder's base -directory now works correctly. The 'buildbot' command tries to create -slightly more useful master/slave directories, adding a Makefile entry to -re-create the .tap file, and removing global-read permissions from the files -that may contain buildslave passwords. - -** twisted-2.0.0 compatibility - -Both buildmaster and buildslave should run properly under Twisted-2.0 . There -are still some warnings about deprecated functions, some of which could be -fixed, but there are others that would require removing compatibility with -Twisted-1.3, and I don't expect to do that until 2.0 has been out and stable -for at least several months. The unit tests should pass under 2.0, whereas -the previous buildbot release had tests which could hang when run against the -new "trial" framework in 2.0. - -The Twisted-specific steps (including Trial) have been updated to match 2.0 -functionality. - -** win32 compatibility - -Thankt to Nick Trout, more compatibility fixes have been incorporated, -improving the chances that the unit tests will pass on windows systems. There -are still some problems, and a step-by-step "running buildslaves on windows" -document would be greatly appreciated. - -** API docs - -Thanks to Thomas Vander Stichele, most of the docstrings have been converted -to epydoc format. There is a utility in docs/gen-reference to turn these into -a tree of cross-referenced HTML pages. Eventually these docs will be -auto-generated and somehow published on the buildbot web page. - - - -* Release 0.6.2 (13 Dec 2004) - -** new features - -It is now possible to interrupt a running build. Both the web page and the -IRC bot feature 'stop build' commands, which can be used to interrupt the -current BuildStep and accelerate the termination of the overall Build. The -status reporting for these still leaves something to be desired (an -'interrupt' event is pushed into the column, and the reason for the interrupt -is added to a pseudo-logfile for the step that was stopped, but if you only -look at the top-level status it appears that the build failed on its own). - -Builds are also halted if the connection to the buildslave is lost. On the -slave side, any active commands are halted if the connection to the -buildmaster is lost. - -** minor new features - -The IRC log bot now reports ETA times in a MMSS format like "2m45s" instead -of the clunky "165 seconds". - -** bug fixes - -*** Slave Disconnect - -Slave disconnects should be handled better now: the current build should be -abandoned properly. Earlier versions could get into weird states where the -build failed to finish, clogging the builder forever (or at least until the -buildmaster was restarted). - -In addition, there are weird network conditions which could cause a -buildslave to attempt to connect twice to the same buildmaster. This can -happen when the slave is sending large logfiles over a slow link, while using -short keepalive timeouts. The buildmaster has been fixed to allow the second -connection attempt to take precedence over the first, so that the older -connection is jettisoned to make way for the newer one. - -In addition, the buildslave has been fixed to be less twitchy about timeouts. -There are now two parameters: keepaliveInterval (which is controlled by the -mktap 'keepalive' argument), and keepaliveTimeout (which requires editing the -.py source to change from the default of 30 seconds). The slave expects to -see *something* from the master at least once every keepaliveInterval -seconds, and will try to provoke a response (by sending a keepalive request) -'keepaliveTimeout' seconds before the end of this interval just in case there -was no regular traffic. Any kind of traffic will qualify, including -acknowledgements of normal build-status updates. - -The net result is that, as long as any given PB message can be sent over the -wire in less than 'keepaliveTimeout' seconds, the slave should not mistakenly -disconnect because of a timeout. There will be traffic on the wire at least -every 'keepaliveInterval' seconds, which is what you want to pay attention to -if you're trying to keep an intervening NAT box from dropping what it thinks -is an abandoned connection. A quiet loss of connection will be detected -within 'keepaliveInterval' seconds. - -*** Large Logfiles - -The web page rendering code has been fixed to deliver large logfiles in -pieces, using a producer/consumer apparatus. This avoids the large spike in -memory consumption when the log file body was linearized into a single string -and then buffered in the socket's application-side transmit buffer. This -should also avoid the 640k single-string limit for web.distrib servers that -could be hit by large (>640k) logfiles. - - - -* Release 0.6.1 (23 Nov 2004) - -** win32 improvements/bugfixes - -Several changes have gone in to improve portability to non-unix systems. It -should be possible to run a build slave under windows without major issues -(although step-by-step documentation is still greatly desired: check the -mailing list for suggestions from current win32 users). - -*** PBChangeSource: use configurable directory separator, not os.sep - -The PBChangeSource, which listens on a TCP socket for change notices -delivered from tools like contrib/svn_buildbot.py, was splitting source -filenames with os.sep . This is inappropriate, because those file names are -coming from the VC repository, not the local filesystem, and the repository -host may be running a different OS (with a different separator convention) -than the buildmaster host. In particular, a win32 buildmaster using a CVS -repository running on a unix box would be confused. - -PBChangeSource now takes a sep= argument to indicate the separator character -to use. - -*** build saving should work better - -windows cannot do the atomic os.rename() trick that unix can, so under win32 -the buildmaster falls back to save/delete-old/rename, which carries a slight -risk of losing a saved build log (if the system were to crash between the -delete-old and the rename). - -** new features - -*** test-result tracking - -Work has begun on fine-grained test-result handling. The eventual goal is to -be able to track individual tests over time, and create problem reports when -a test starts failing (which then are resolved when the test starts passing -again). The first step towards this is an ITestResult interface, and code in -the TrialTestParser to create such results for all non-passing tests (the -ones for which Trial emits exception tracebacks). - -These test results are currently displayed in a tree-like display in a page -accessible from each Build's page (follow the numbered link in the yellow -box at the start of each build to get there). - -This interface is still in flux, as it really wants to be able to accomodate -things like compiler warnings and tests that are skipped because of missing -libraries or unsupported architectures. - -** bug fixes - -*** VC updates should survive temporary failures - -Some VC systems (CVS and SVN in particular) get upset when files are turned -into directories or vice versa, or when repository items are moved without -the knowledge of the VC system. The usual symptom is that a 'cvs update' -fails where a fresh checkout succeeds. - -To avoid having to manually intervene, the build slaves' VC commands have -been refactored to respond to update failures by deleting the tree and -attempting a full checkout. This may cause some unnecessary effort when, -e.g., the CVS server falls off the net, but in the normal case it will only -come into play when one of these can't-cope situations arises. - -*** forget about an existing build when the slave detaches - -If the slave was lost during a build, the master did not clear the -.currentBuild reference, making that builder unavailable for later builds. -This has been fixed, so that losing a slave should be handled better. This -area still needs some work, I think it's still possible to get both the -slave and the master wedged by breaking the connection at just the right -time. Eventually I want to be able to resume interrupted builds (especially -when the interruption is the result of a network failure and not because the -slave or the master actually died). - -*** large logfiles now consume less memory - -Build logs are stored as lists of (type,text) chunks, so that -stdout/stderr/headers can be displayed differently (if they were -distinguishable when they were generated: stdout and stderr are merged when -usePTY=1). For multi-megabyte logfiles, a large list with many short strings -could incur a large overhead. The new behavior is to merge same-type string -chunks together as they are received, aiming for a chunk size of about 10kb, -which should bring the overhead down to a more reasonable level. - -There remains an issue with actually delivering large logfiles over, say, -the HTML interface. The string chunks must be merged together into a single -string before delivery, which causes a spike in the memory usage when the -logfile is viewed. This can also break twisted.web.distrib -type servers, -where the underlying PB protocol imposes a 640k limit on the size of -strings. This will be fixed (with a proper Producer/Consumer scheme) in the -next release. - - -* Release 0.6.0 (30 Sep 2004) - -** new features - -*** /usr/bin/buildbot control tool - -There is now an executable named 'buildbot'. For now, this just provides a -convenient front-end to mktap/twistd/kill, but eventually it will provide -access to other client functionality (like the 'try' builds, and a status -client). Assuming you put your buildbots in /var/lib/buildbot/master/FOO, -you can do 'buildbot create-master /var/lib/buildbot/master/FOO' and it will -create the .tap file and set up a sample master.cfg for you. Later, -'buildbot start /var/lib/buildbot/master/FOO' will start the daemon. - - -*** build status now saved in external files, -shutdown.tap unnecessary - -The status rewrite included a change to save all build status in a set of -external files. These files, one per build, are put in a subdirectory of the -master's basedir (named according to the 'builddir' parameter of the Builder -configuration dictionary). This helps keep the buildmaster's memory -consumption small: the (potentially large) build logs are kept on disk -instead of in RAM. There is a small cache (2 builds per builder) kept in -memory, but everything else lives on disk. - -The big change is that the buildmaster now keeps *all* status in these -files. It is no longer necessary to preserve the buildbot-shutdown.tap file -to run a persistent buildmaster. The buildmaster may be launched with -'twistd -f buildbot.tap' each time, in fact the '-n' option can be added to -prevent twistd from automatically creating the -shutdown.tap file. - -There is still one lingering bug with this change: the Expectations object -for each builder (which records how long the various steps took, to provide -an ETA value for the next time) is not yet saved. The result is that the -first build after a restart will not provide an ETA value. - -0.6.0 keeps status in a single file per build, as opposed to 0.5.0 which -kept status in many subdirectories (one layer for builds, another for steps, -and a third for logs). 0.6.0 will detect and delete these subdirectories as -it overwrites them. - -The saved builds are optional. To prevent disk usage from growing without -bounds, you may want to set up a cron job to run 'find' and delete any which -are too old. The status displays will happily survive without those saved -build objects. - -The set of recorded Changes is kept in a similar file named 'changes.pck'. - - -*** source checkout now uses timestamp/revision - -Source checkouts are now performed with an appropriate -D TIMESTAMP (for -CVS) or -r REVISION (for SVN) marker to obtain the exact sources that were -specified by the most recent Change going into the current Build. This -avoids a race condition in which a change might be committed after the build -has started but before the source checkout has completed, resulting in a -mismatched set of source files. Such changes are now ignored. - -This works by keeping track of repository-wide revision/transaction numbers -(for version control systems that offer them, like SVN). The checkout or -update is performed with the highest such revision number. For CVS (which -does not have them), the timestamp of each commit message is used, and a -D -argument is created to place the checkout squarely in the middle of the "tree -stable timer"'s window. - -This also provides the infrastructure for the upcoming 'try' feature. All -source-checkout commands can now obtain a base revision marker and a patch -from the Build, allowing certain builds to be performed on something other -than the most recent sources. - -See source.xhtml and steps.xhtml for details. - - -*** Darcs and Arch support added - -There are now build steps which retrieve a source tree from Darcs and Arch -repositories. See steps.xhtml for details. - -Preliminary P4 support has been added, thanks to code from Dave Peticolas. -You must manually set up each build slave with an appropriate P4CLIENT: all -buildbot does is run 'p4 sync' at the appropriate times. - - -*** Status reporting rewritten - -Status reporting was completely revamped. The config file now accepts a -BuildmasterConfig['status'] entry, with a list of objects that perform status -delivery. The old config file entries which controlled the web status port -and the IRC bot have been deprecated in favor of adding instances to -['status']. The following status-delivery classes have been implemented, all -in the 'buildbot.status' package: - - client.PBListener(port, username, passwd) - html.Waterfall(http_port, distrib_port) - mail.MailNotifier(fromaddr, mode, extraRecipients..) - words.IRC(host, nick, channels) - -See the individual docstrings for details about how to use each one. You can -create new status-delivery objects by following the interfaces found in the -buildbot.interfaces module. - - -*** BuildFactory configuration process changed - -The basic BuildFactory class is now defined in buildbot.process.factory -rather than buildbot.process.base, so you will have to update your config -files. factory.BuildFactory is the base class, which accepts a list of Steps -to run. See docs/factories.xhtml for details. - -There are now easier-to-use BuildFactory classes for projects which use GNU -Autoconf, perl's MakeMaker (CPAN), python's distutils (but no unit tests), -and Twisted's Trial. Each one takes a separate 'source' Step to obtain the -source tree, and then fills in the rest of the Steps for you. - - -*** CVS/SVN VC steps unified, simplified - -The confusing collection of arguments for the CVS step ('clobber=', -'copydir=', and 'export=') have been removed in favor of a single 'mode' -argument. This argument describes how you want to use the sources: whether -you want to update and compile everything in the same tree (mode='update'), -or do a fresh checkout and full build each time (mode='clobber'), or -something in between. - -The SVN (Subversion) step has been unified and accepts the same mode= -parameter as CVS. New version control steps will obey the same interface. - -Most of the old configuration arguments have been removed. You will need to -update your configuration files to use the new arguments. See -docs/steps.xhtml for a description of all the new parameters. - - -*** Preliminary Debian packaging added - -Thanks to the contributions of Kirill Lapshin, we can now produce .deb -installer packages. These are still experimental, but they include init.d -startup/shutdown scripts, which the the new /usr/bin/buildbot to invoke -twistd. Create your buildmasters in /var/lib/buildbot/master/FOO, and your -slaves in /var/lib/buildbot/slave/BAR, then put FOO and BAR in the -appropriate places in /etc/default/buildbot . After that, the buildmasters -and slaves will be started at every boot. - -Pre-built .debs are not yet distributed. Use 'debuild -uc -us' from the -source directory to create them. - - -** minor features - - -*** Source Stamps - -Each build now has a "source stamp" which describes what sources it used. The -idea is that the sources for this particular build can be completely -regenerated from the stamp. The stamp is a tuple of (revision, patch), where -the revision depends on the VC system being used (for CVS it is either a -revision tag like "BUILDBOT-0_5_0" or a datestamp like "2004/07/23", for -Subversion it is a revision number like 11455). This must be combined with -information from the Builder that is constant across all builds (something to -point at the repository, and possibly a branch indicator for CVS and other VC -systems that don't fold this into the repository string). - -The patch is an optional unified diff file, ready to be applied by running -'patch -p0 <PATCH' from inside the workdir. This provides support for the -'try' feature that will eventually allow developers to run buildbot tests on -their code before checking it in. - - -*** SIGHUP causes the buildmaster's configuration file to be re-read - -*** IRC bot now has 'watch' command - -You can now tell the buildbot's IRC bot to 'watch <buildername>' on a builder -which is currently performing a build. When that build is finished, the -buildbot will make an announcement (including the results of the build). - -The IRC 'force build' command will also announce when the resulting build has -completed. - - -*** the 'force build' option on HTML and IRC status targets can be disabled - -The html.Waterfall display and the words.IRC bot may be constructed with an -allowForce=False argument, which removes the ability to force a build through -these interfaces. Future versions will be able to restrict this build-forcing -capability to authenticated users. The per-builder HTML page no longer -displays the 'Force Build' buttons if it does not have this ability. Thanks -to Fred Drake for code and design suggestions. - - -*** master now takes 'projectName' and 'projectURL' settings - -These strings allow the buildbot to describe what project it is working for. -At the moment they are only displayed on the Waterfall page, but in the next -release they will be retrieveable from the IRC bot as well. - - -*** survive recent (SVN) Twisted versions - -The buildbot should run correctly (albeit with plenty of noisy deprecation -warnings) under the upcoming Twisted-2.0 release. - - -*** work-in-progress realtime Trial results acquisition - -Jonathan Simms (<slyphon>) has been working on 'retrial', a rewrite of -Twisted's unit test framework that will most likely be available in -Twisted-2.0 . Although it is not yet complete, the buildbot will be able to -use retrial in such a way that build status is reported on a per-test basis, -in real time. This will be the beginning of fine-grained test tracking and -Problem management, described in docs/users.xhtml . - - -* Release 0.5.0 (22 Jul 2004) - -** new features - -*** web.distrib servers via TCP - -The 'webPathname' config option, which specifies a UNIX socket on which to -publish the waterfall HTML page (for use by 'mktap web -u' or equivalent), -now accepts a numeric port number. This publishes the same thing via TCP, -allowing the parent web server to live on a separate machine. - -This config option could be named better, but it will go away altogether in -a few releases, when status delivery is unified. It will be replaced with a -WebStatusTarget object, and the config file will simply contain a list of -various kinds of status targets. - -*** 'master.cfg' filename is configurable - -The buildmaster can use a config file named something other than -"master.cfg". Use the --config=foo.cfg option to mktap to control this. - -*** FreshCVSSource now uses newcred (CVSToys >= 1.0.10) - -The FreshCVSSource class now defaults to speaking to freshcvs daemons from -modern CVSToys releases. If you need to use the buildbot with a daemon from -CVSToys-1.0.9 or earlier, use FreshCVSSourceOldcred instead. Note that the -new form only requires host/port/username/passwd: the "serviceName" -parameter is no longer meaningful. - -*** Builders are now configured with a dictionary, not a tuple - -The preferred way to set up a Builder in master.cfg is to provide a -dictionary with various keys, rather than a (non-extensible) 4-tuple. See -docs/config.xhtml for details. The old tuple-way is still supported for now, -it will probably be deprecated in the next release and removed altogether in -the following one. - -*** .periodicBuildTime is now exposed to the config file - -To set a builder to run at periodic intervals, simply add a -'periodicBuildTime' key to its master.cfg dictionary. Again, see -docs/config.xhtml for details. - -*** svn_buildbot.py adds --include, --exclude - -The commit trigger script now gives you more control over which files are -sent to the buildmaster and which are not. - -*** usePTY is controllable at slave mktap time - -The buildslaves usually run their child processes in a pty, which creates a -process group for all the children, which makes it much easier to kill them -all at once (i.e. if a test hangs). However this causes problems on some -systems. Rather than hacking slavecommand.py to disable the use of these -ptys, you can now create the slave's .tap file with --usepty=0 at mktap -time. - -** Twisted changes - -A summary of warnings (e.g. DeprecationWarnings) is provided as part of the -test-case summarizer. The summarizer also counts Skips, expectedFailures, -and unexpectedSuccesses, displaying the counts on the test step's event box. - -The RunUnitTests step now uses "trial -R twisted" instead of "trial -twisted.test", which is a bit cleaner. All .pyc files are deleted before -starting trial, to avoid getting tripped up by deleted .py files. - -** documentation - -docs/config.xhtml now describes the syntax and allowed contents of the -'master.cfg' configuration file. - -** bugfixes - -Interlocks had a race condition that could cause the lock to get stuck -forever. - -FreshCVSSource has a prefix= argument that was moderately broken (it used to -only work if the prefix was a single directory component). It now works with -subdirectories. - -The buildmaster used to complain when it saw the "info" directory in a -slave's workspace. This directory is used to publish information about the -slave host and its administrator, and is not a leftover build directory as -the complaint suggested. This complain has been silenced. - - -* Release 0.4.3 (30 Apr 2004) - -** PBChangeSource made explicit - -In 0.4.2 and before, an internal interface was available which allowed -special clients to inject changes into the Buildmaster. This interface is -used by the contrib/svn_buildbot.py script. The interface has been extracted -into a proper PBChangeSource object, which should be created in the -master.cfg file just like the other kinds of ChangeSources. See -docs/sources.xhtml for details. - -If you were implicitly using this change source (for example, if you use -Subversion and the svn_buildbot.py script), you *must* add this source to -your master.cfg file, or changes will not be delivered and no builds will be -triggered. - -The PBChangeSource accepts the same "prefix" argument as all other -ChangeSources. For a SVN repository that follows the recommended practice of -using "trunk/" for the trunk revisions, you probably want to construct the -source like this: - - source = PBChangeSource(prefix="trunk") - -to make sure that the Builders are given sensible (trunk-relative) -filenames for each changed source file. - -** Twisted changes - -*** step_twisted.RunUnitTests can change "bin/trial" - -The twisted RunUnitTests step was enhanced to let you run something other -than "bin/trial", making it easier to use a buildbot on projects which use -Twisted but aren't actually Twisted itself. - -*** Twisted now uses Subversion - -Now that Twisted has moved from CVS to SVN, the Twisted build processes have -been modified to perform source checkouts from the Subversion repository. - -** minor feature additions - -*** display Changes with HTML - -Changes are displayed with a bit more pizazz, and a links= argument was -added to allow things like ViewCVS links to be added to the display -(although it is not yet clear how this argument should be used: the -interface remains subject to change untill it has been documented). - -*** display ShellCommand logs with HTML - -Headers are in blue, stderr is in red (unless usePTY=1 in which case stderr -and stdout are indistinguishable). A link is provided which returns the same -contents as plain text (by appending "?text=1" to the URL). - -*** buildslaves send real tracebacks upon error - -The .unsafeTracebacks option has been turned on for the buildslaves, -allowing them to send a full stack trace when an exception occurs, which is -logged in the buildmaster's twistd.log file. This makes it much easier to -determine what went wrong on the slave side. - -*** BasicBuildFactory refactored - -The BasicBuildFactory class was refactored to make it easier to create -derivative classes, in particular the BasicSVN variant. - -*** "ping buildslave" web button added - -There is now a button on the "builder information" page that lets a web user -initiate a ping of the corresponding build slave (right next to the button -that lets them force a build). This was added to help track down a problem -with the slave keepalives. - -** bugs fixed: - -You can now have multiple BuildSteps with the same name (the names are used -as hash keys in the data structure that helps determine ETA values for each -step, the new code creates unique key names if necessary to avoid -collisions). This means that, for example, you do not have to create a -BuildStep subclass just to have two Compile steps in the same process. - -If CVSToys is not installed, the tests that depend upon it are skipped. - -Some tests in 0.4.2 failed because of a missing set of test files, they are -now included in the tarball properly. - -Slave keepalives should work better now in the face of silent connection -loss (such as when an intervening NAT box times out the association), the -connection should be reestablished in minutes instead of hours. - -Shell commands on the slave are invoked with an argument list instead of the -ugly and error-prone split-on-spaces approach. If the ShellCommand is given -a string (instead of a list), it will fall back to splitting on spaces. -Shell commands should work on win32 now (using COMSPEC instead of /bin/sh). - -Buildslaves under w32 should theoretically work now, and one was running for -the Twisted buildbot for a while until the machine had to be returned. - -The "header" lines in ShellCommand logs (which include the first line, that -displays the command being run, and the last, which shows its exit status) -are now generated by the buildslave side instead of the local (buildmaster) -side. This can provide better error handling and is generally cleaner. -However, if you have an old buildslave (running 0.4.2 or earlier) and a new -buildmaster, then neither end will generate these header lines. - -CVSCommand was improved, in certain situations 0.4.2 would perform -unnecessary checkouts (when an update would have sufficed). Thanks to Johan -Dahlin for the patches. The status output was fixed as well, so that -failures in CVS and SVN commands (such as not being able to find the 'svn' -executable) make the step status box red. - -Subversion support was refactored to make it behave more like CVS. This is a -work in progress and will be improved in the next release. - - -* Release 0.4.2 (08 Jan 2004) - -** test suite updated - -The test suite has been completely moved over to Twisted's "Trial" -framework, and all tests now pass. To run the test suite (consisting of 64 -tests, probably covering about 30% of BuildBot's logic), do this: - - PYTHONPATH=. trial -v buildbot.test - -** Mail parsers updated - -Several bugs in the mail-parsing code were fixed, allowing a buildmaster to -be triggered by mail sent out by a CVS repository. (The Twisted Buildbot is -now using this to trigger builds, as their CVS server machine is having some -difficulties with FreshCVS). The FreshCVS mail format for directory -additions appears to have changed recently: the new parser should handle -both old and new-style messages. - -A parser for Bonsai commit messages (buildbot.changes.mail.parseBonsaiMail) -was contributed by Stephen Davis. Thanks Stephen! - -** CVS "global options" now available - -The CVS build step can now accept a list of "global options" to give to the -cvs command. These go before the "update"/"checkout" word, and are described -fully by "cvs --help-options". Two useful ones might be "-r", which causes -checked-out files to be read-only, and "-R", which assumes the repository is -read-only (perhaps by not attempting to write to lock files). - - -* Release 0.4.1 (09 Dec 2003) - -** MaildirSources fixed - -Several bugs in MaildirSource made them unusable. These have been fixed (for -real this time). The Twisted buildbot is using an FCMaildirSource while they -fix some FreshCVS daemon problems, which provided the encouragement for -getting these bugs fixed. - -In addition, the use of DNotify (only available under linux) was somehow -broken, possibly by changes in some recent version of Python. It appears to -be working again now (against both python-2.3.3c1 and python-2.2.1). - -** master.cfg can use 'basedir' variable - -As documented in the sample configuration file (but not actually implemented -until now), a variable named 'basedir' is inserted into the namespace used -by master.cfg . This can be used with something like: - - os.path.join(basedir, "maildir") - -to obtain a master-basedir-relative location. - - -* Release 0.4.0 (05 Dec 2003) - -** newapp - -I've moved the codebase to Twisted's new 'application' framework, which -drastically cleans up service startup/shutdown just like newcred did for -authorization. This is mostly an internal change, but the interface to -IChangeSources was modified, so in the off chance that someone has written a -custom change source, it may have to be updated to the new scheme. - -The most user-visible consequence of this change is that now both -buildmasters and buildslaves are generated with the standard Twisted 'mktap' -utility. Basic documentation is in the README file. - -Both buildmaster and buildslave .tap files need to be re-generated to run -under the new code. I have not figured out the styles.Versioned upgrade path -well enough to avoid this yet. Sorry. - -This also means that both buildslaves and the buildmaster require -Twisted-1.1.0 or later. - -** reloadable master.cfg - -Most aspects of a buildmaster is now controlled by a configuration file -which can be re-read at runtime without losing build history. This feature -makes the buildmaster *much* easier to maintain. - -In the previous release, you would create the buildmaster by writing a -program to define the Builders and ChangeSources and such, then run it to -create the .tap file. In the new release, you use 'mktap' to create the .tap -file, and the only parameter you give it is the base directory to use. Each -time the buildmaster starts, it will look for a file named 'master.cfg' in -that directory and parse it as a python script. That script must define a -dictionary named 'BuildmasterConfig' with various keys to define the -builders, the known slaves, what port to use for the web server, what IRC -channels to connect to, etc. - -This config file can be re-read at runtime, and the buildmaster will compute -the differences and add/remove services as necessary. The re-reading is -currently triggered through the debug port (contrib/debugclient.py is the -debug port client), but future releases will add the ability to trigger the -reconfiguration by IRC command, web page button, and probably a local UNIX -socket (with a helper script to trigger a rebuild locally). - -docs/examples/twisted_master.cfg contains a sample configuration file, which -also lists all the keys that can be set. - -There may be some bugs lurking, such as re-configuring the buildmaster while -a build is running. It needs more testing. - -** MaxQ support - -Radix contributed some support scripts to run MaxQ test scripts. MaxQ -(http://maxq.tigris.org/) is a web testing tool that allows you to record -HTTP sessions and play them back. - -** Builders can now wait on multiple Interlocks - -The "Interlock" code has been enhanced to allow multiple builders to wait on -each one. This was done to support the new config-file syntax for specifying -Interlocks (in which each interlock is a tuple of A and [B], where A is the -builder the Interlock depends upon, and [B] is a list of builders that -depend upon the Interlock). - -"Interlock" is misnamed. In the next release it will be changed to -"Dependency", because that's what it really expresses. A new class (probably -called Interlock) will be created to express the notion that two builders -should not run at the same time, useful when multiple builders are run on -the same machine and thrashing results when several CPU- or disk- intensive -compiles are done simultaneously. - -** FreshCVSSource can now handle newcred-enabled FreshCVS daemons - -There are now two FreshCVSSource classes: FreshCVSSourceNewcred talks to -newcred daemons, and FreshCVSSourceOldcred talks to oldcred ones. Mind you, -FreshCVS doesn't yet do newcred, but when it does, we'll be ready. - -'FreshCVSSource' maps to the oldcred form for now. That will probably change -when the current release of CVSToys supports newcred by default. - -** usePTY=1 on posix buildslaves - -When a buildslave is running under POSIX (i.e. pretty much everything except -windows), child processes are created with a pty instead of separate -stdin/stdout/stderr pipes. This makes it more likely that a hanging build -(when killed off by the timeout code) will have all its sub-childred cleaned -up. Non-pty children would tend to leave subprocesses running because the -buildslave was only able to kill off the top-level process (typically -'make'). - -Windows doesn't have any concept of ptys, so non-posix systems do not try to -enable them. - -** mail parsers should actually work now - -The email parsing functions (FCMaildirSource and SyncmailMaildirSource) were -broken because of my confused understanding of how python class methods -work. These sources should be functional now. - -** more irc bot sillyness - -The IRC bot can now perform half of the famous AYBABTO scene. - - -* Release 0.3.5 (19 Sep 2003) - -** newcred - -Buildbot has moved to "newcred", a new authorization framework provided by -Twisted, which is a good bit cleaner and easier to work with than the -"oldcred" scheme in older versions. This causes both buildmaster and -buildslaves to depend upon Twisted 1.0.7 or later. The interface to -'makeApp' has changed somewhat (the multiple kinds of remote connections all -use the same TCP port now). - -Old buildslaves will get "_PortalWrapper instance has no attribute -'remote_username'" errors when they try to connect. They must be upgraded. - -The FreshCVSSource uses PB to connect to the CVSToys server. This has been -upgraded to use newcred too. If you get errors (TODO: what do they look -like?) in the log when the buildmaster tries to connect, you need to upgrade -your FreshCVS service or use the 'useOldcred' argument when creating your -FreshCVSSource. This is a temporary hack to allow the buildmaster to talk to -oldcred CVSToys servers. Using it will trigger deprecation warnings. It will -go away eventually. - -In conjunction with this change, makeApp() now accepts a password which can -be applied to the debug service. - -** new features - -*** "copydir" for CVS checkouts - -The CVS build step can now accept a "copydir" parameter, which should be a -directory name like "source" or "orig". If provided, the CVS checkout is -done once into this directory, then copied into the actual working directory -for compilation etc. Later updates are done in place in the copydir, then -the workdir is replaced with a copy. - -This reduces CVS bandwidth (update instead of full checkout) at the expense -of twice the disk space (two copies of the tree). - -*** Subversion (SVN) support - -Radix (Christopher Armstrong) contributed early support for building -Subversion-based trees. The new 'SVN' buildstep behaves roughly like the -'CVS' buildstep, and the contrib/svn_buildbot.py script can be used as a -checkin trigger to feed changes to a running buildmaster. - -** notable bugfixes - -*** .tap file generation - -We no longer set the .tap filename, because the buildmaster/buildslave -service might be added to an existing .tap file and we shouldn't presume to -own the whole thing. You may want to manually rename the "buildbot.tap" file -to something more meaningful (like "buildslave-bot1.tap"). - -*** IRC reconnect - -If the IRC server goes away (it was restarted, or the network connection was -lost), the buildmaster will now schedule a reconnect attempt. - -*** w32 buildslave fixes - -An "rm -rf" was turned into shutil.rmtree on non-posix systems. - - -* Release 0.3.4 (28 Jul 2003) - -** IRC client - -The buildmaster can now join a set of IRC channels and respond to simple -queries about builder status. - -** slave information - -The build slaves can now report information from a set of info/* files in -the slave base directory to the buildmaster. This will be used by the slave -administrator to announce details about the system hosting the slave, -contact information, etc. For now, info/admin should contain the name/email -of the person who is responsible for the buildslave, and info/host should -describe the system hosting the build slave (OS version, CPU speed, memory, -etc). The contents of these files are made available through the waterfall -display. - -** change notification email parsers - -A parser for Syncmail (syncmail.sourceforge.net) was added. SourceForge -provides examples of setting up syncmail to deliver CVS commit messages to -mailing lists, so hopefully this will make it easier for sourceforge-hosted -projects to set up a buildbot. - -email processors were moved into buildbot.changes.mail . FCMaildirSource was -moved, and the compatibility location (buildbot.changes.freshcvsmail) will -go away in the next release. - -** w32 buildslave ought to work - -Some non-portable code was changed to make it more likely that the -buildslave will run under windows. The Twisted buildbot now has a -(more-or-less) working w32 buildslave. - - -* Release 0.3.3 (21 May 2003): - -** packaging changes - -*** include doc/examples in the release. Oops again. - -** network changes - -*** add keepalives to deal with NAT boxes - -Some NAT boxes drop port mappings if the TCP connection looks idle for too -long (maybe 30 minutes?). Add application-level keepalives (dummy commands -sent from slave to master every 10 minutes) to appease the NAT box and keep -our connection alive. Enable this with --keepalive in the slave mktap -command line. Check the README for more details. - -** UI changes - -*** allow slaves to trigger any build that they host - -Added an internal function to ask the buildmaster to start one of their -builds. Must be triggered with a debugger or manhole on the slave side for -now, will add a better UI later. - -*** allow web page viewers to trigger any build - -Added a button to the per-build page (linked by the build names on the third -row of the waterfall page) to allow viewers to manually trigger builds. -There is a field for them to indicate who they are and why they are -triggering the build. It is possible to abuse this, but for now the benefits -outweigh the damage that could be done (worst case, someone can make your -machine run builds continuously). - -** generic buildprocess changes - -*** don't queue multiple builds for offline slaves - -If a slave is not online when a build is ready to run, that build is queued -so the slave will run it when it next connects. However, the buildmaster -used to queue every such build, so the poor slave machine would be subject -to tens or hundreds of builds in a row when they finally did come online. -The buildmaster has been changed to merge these multiple builds into a -single one. - -*** bump ShellCommand default timeout to 20 minutes - -Used for testing out the win32 twisted builder. I will probably revert this -in the next relese. - -*** split args in ShellCommand ourselves instead of using /bin/sh - -This should remove the need for /bin/sh on the slave side, improving the -chances that the buildslave can run on win32. - -*** add configureEnv argument to Configure step, pass env dict to slave - -Allows build processes to do things like 'CFLAGS=-O0 ./configure' without -using /bin/sh to set the environment variable - -** Twisted buildprocess changes - -*** warn instead of flunk the build when cReactor or qtreactor tests fail - -These two always fail. For now, downgrade those failures to a warning -(orange box instead of red). - -*** don't use 'clobber' on remote builds - -Builds that run on remote machines (freebsd, OS-X) now use 'cvs update' -instead of clobbering their trees and doing a fresh checkout. The multiple -simultaneous CVS checkouts were causing a strain on Glyph's upstream -bandwidth. - -*** use trial --testmodule instead of our own test-case-name grepper - -The Twisted coding/testing convention has developers put 'test-case-name' -tags (emacs local variables, actually) in source files to indicate which -test cases should be run to exercise that code. Twisted's unit-test -framework just acquired an argument to look for these tags itself. Use that -instead of the extra FindUnitTestsForFiles build step we were doing before. -Removes a good bit of code from buildbot and into Twisted where it really -belongs. - - -* Release 0.3.2 (07 May 2003): - -** packaging changes - -*** fix major packaging bug: none of the buildbot/* subdirectories were -included in the 0.3.1 release. Sorry, I'm still figuring out distutils -here.. - -** internal changes - -*** use pb.Cacheable to update Events in remote status client. much cleaner. - -*** start to clean up BuildProcess->status.builder interface - -** bug fixes - -*** waterfall display was missing a <tr>, causing it to be misrendered in most -browsers (except the one I was testing it with, of course) - -*** URL without trailing slash (when served in a twisted-web distributed -server, with a url like "http://twistedmatrix.com/~warner.twistd") should do -redirect to URL-with-trailing-slash, otherwise internal hrefs are broken. - -*** remote status clients: forget RemoteReferences at shutdown, removes -warnings about "persisting Ephemerals" - -** Twisted buildprocess updates: - -*** match build process as of twisted-1.0.5 -**** use python2.2 everywhere now that twisted rejects python2.1 -**** look for test-result constants in multiple places -*** move experimental 'trial --jelly' code to separate module -*** add FreeBSD builder -*** catch rc!=0 in HLint step -*** remove RunUnitTestsRandomly, use randomly=1 parameter instead -*** parameterize ['twisted.test'] default test case to make subclassing easier -*** ignore internal distutils warnings in python2.3 builder - - -* Release 0.3.1 (29 Apr 2003): - -** First release. - -** Features implemented: - - change notification from FreshCVS server or parsed maildir contents - - timed builds - - basic builds, configure/compile/test - - some Twisted-specific build steps: docs, unit tests, debuild - - status reporting via web page - -** Features still experimental/unpolished - - status reporting via PB client |