[Testing] Buildbot Questions

[Grig Gheorghiu]

Hi, Michael

I'm finally getting around to answering some of your questions. See my
answers inline.

--- Michael Stone <michael at laptop.org> wrote:

> Titus, Grig,
> 
> I have some questions about the proper usage of Buildbot and about
> how
> the current design for pybots.org originated. The design questions
> should definitely be considered to be in dialogue with Titus'
> two-month-old post:
> 
>  
>
http://ivory.idyll.org/blog/oct-07/pull-vs-push-continuous-integration.html
> 
>
------------------------------------------------------------------------
> 
> First, I've made a buildmaster and I've equipped it with a
> PilgrimChangeSource which polls the website where new builds appear
> and
> which creates Changes with appropriate hyperlinks when they show up.
> 
> Implementation Question 1: How (or maybe "where") should I calculate
> the
> URLs for the build tarball from the knowledge that, say, build
> 'joyride-1384' is available?
> 
>   At present, I'm performing this calculation in my
> PilgrimChangeSource
>   and I'm encoding the information in two new attributes in the
> Change
>   object I create in that ChangeSource.
> 
>   Perhaps this logic would be more appropriately placed in a
>   PilgrimChange object? Or somewhere farther "downstream" from the
>   ChangeSource?

To be honest, I never had to create ChangeSource objects in my buildbot
configurations. I think this question will be better answered on the
buildbot-dev mailing list. Whenever I set up buildbot, the change
notifications I used was the default one, sent by the source control
system (CVS or SVN usually). But I can see how in this case you need a
custom change notification.


> 
> Implementation Question 2: Given the Design Questions which follow,
> what
> are some decent ways to proceed in implementing my link-checker and
> permissions-checker bots?
> 
>   In particular, should I concentrate on writing specific
>   BuildSteps/RemoteCommands that describe the appropriate operations
> or
>   should I work on writing a single 'ChrootTest' buildstep that my
>   test-bots can implement, each in their own way?
> 


>
------------------------------------------------------------------------
> 
> Design Question 1: Why does buildbot use a master-slave architecture?
> 
>   Our build process has several steps that must be run as root on
> fairly
>   carefully configured F-7 machines. Since buildbot seems to be set
> up
>   to operate by granting remote shell access to the build master, we
>   would effectively be granting the buildmaster root access to every
>   build-slave on which we actually try to build images. This is not
> so
>   happy, particularly given the fact that BuildBot transmits
> plaintext
>   authentication credentials.

In Titus's terminology, builbot uses a 'pull' mechanism, with a central
master coordinating the work of a farm of slaves and pulling the build
results from them. This offers the advantage of running the build
immediately on all slaves, as soon as the master is notified by a
change. The alternative (embraced by Tinderbox for example) is to let
each slave do its own thing, at its own pace, and then send the results
to the master. IMO the builbot approach is better because it gives you
instant feedback across your slave server farm. 

The master doesn't need remote shell access on the slaves. It
communicates with the slaves via sockets. All the master does is send
build commands to the slaves. The slaves then execute these commands,
then report back the results to the master. The master is then
responsible with aggregating the results in a nice status page.

Both the master process and the slave process can run as non-root users
(in fact it's recommended that they do so).

> 
> Design Question 2: Why is the information about how to perform builds
> and about how to interpret the results centralized on the
> buildmaster?
> 
>   We have had serious problems with human bottlenecks and with
>   machines/systems that only one person knows how to administer. Are
>   there examples of how to use buildbot a fashion that distributes
>   maintenance reponsibility, for example, by distributing the action
> of
>   adding and configuring new buildslaves?
> 

In my experience, you do need a person appointed as the 'human build
master' who will be in charge of the buildbot master configuration
file. There is no easy way to distribute the action of modifying that
all-important file.


> Design Question 3: Why does the pybots design drop the problems of
> test-setup and test-dependency-satisfaction into the
> 'object-language'
> of shell commands and BuildStep-ordering instead of placing it in the
> 'meta-language' of buildbot abstractions (like BuildSteps, Builds,
> ...)?
> 
>   We have some wildly different kinds of tests that we might want to
>   collectively control and from which we might want to merge status
> info.
>   These include chroot-only tests, inside-QEMU tests,
> on-several-laptop
>   mesh tests, and on-physically-modified-laptop power-management
> stuff.
>   What are some of the tradeoffs between an on-slave opaque
>   'prepare-pkgs.sh' vs. an on-slave 'standard Makefile'-like solution
> vs.
>   a meta-language 'hierarchy-of-builders' solution?
> 

In the pybots example, we preferred to standardize the operation of the
buildslaves by having all of them implement the same shell command. It
could just as easily have been a standard 'make test' command. 

One advantage to having predetermined build steps sent to each slave is
that you can see a PASS/FAIL status clearly for each step. If you let
the slave do a lot of work inside a generic 'make test' command, you'll
just see the aggregate PASS/FAIL for this command, and not for the
particular sub-tasks that it includes. 

Grig