[Testing] Some progress on infrastructure.

[Michael Stone]

Grig,

I spent this afternoon reading through the buildbot manual and it looks
like it might be adaptable to the shape of my needs. Here's how I intend
to begin:

1) Since I expect rapid growth in the quantity and diversity of
   measurements being made and configurations being analyzed, an
   architecture like that of pybots seems appropriate. Therefore, I'd
   like to set up just the minimal buildbot configuration required to
   distribute the checking of exactly two assertions in response to
   build completion. 

   In the future, I'll be happy to investigate running our OS image
   build tools under buildbot for even better status reporting,
   change/failure correlation, and performance.

2) The assertions I want to check are that 

   * /home/olpc is owned by olpc/olpc with mode 0775 
   * /usr/share/activities/*/build_index.html is writable by group olpc

   These properties can be checked directly on the filesystem tarball;
   no fancy setup, teardown, or environment is necessary to collect or
   to interpret the results.

   In the future, I will clearly want to check properties that are
   harder to measure. 

3) Once I get this far, it will hopefully be obvious how to proceed
   further.

Sound good?

Michael


On Wed, Nov 28, 2007 at 10:47:04AM -0800, Grig Gheorghiu wrote:
> My take on this is that you need the infrastructure in place that runs
> automated tests every X hours, or upon every checkin to a given OLPC
> sub-project. Once you get that running, it should be fairly easy for
> people to drop their tests in there. I think you could use very
> successfully a setup similar to the one of Pybots, where volunteers can
> donate buildslaves that run automated tests upon checkins to the Sugar
> API for example. Of course, measuring coverage with figleaf is the
> natural next step.