[Testing] Testing summary - 6 March 2010, Auckland NZ

[James Cameron]

On Sat, Mar 06, 2010 at 01:56:27PM +1300, Tabitha Roder wrote:
> Tom played with olpc tracker and we discussed how best to manage
> community testing.

I've had a think about this.  Different types of testing can happen;

1.  reproducing or further isolating an already reported problem,
especially one that is difficult to reproduce already, for which we use
next action "reproduce" in OLPC trac,

2.  testing following change, in particular testing of what was supposed
to have been changed, for which we use next action "test in build" in
OLPC trac,

3.  testing following integration of a change in a release, again testing
of what was supposed to have been changed, but also how that change is
integrated with other components, for which we use next action "test in
release" in OLPC trac,

4.  testing separate to the change process, such as the testing of
specific activities or Sugar versions.  It is this type of testing that
I see you are concentrating on.

As for the management of community testing, there's a few things that I
can see as requirements:

a.  coverage; ensure that everything is tested,

b.  redundancy; ensure that components are tested by enough people,

c.  duplication; ensure that volunteers are testing things that actually
need testing, and not testing things that others have tested already,

d.  recording; showing a history of testing, so as to feed in to
coverage, and show progress.

> There is a "Next Action" field on the dev.laptop.org tracker with a
> "test in build" option, but there are a lot of different tickets with
> this action, many of which aren't suitable, ...

Yes.  Most of those are part of engineering process testing.  Use either
a ticket type of "task" or keywords, and adjust the query accordingly.

> We propose creating a "community testing" value for this field and
> using the http://dev.laptop.org trac. However, we mostly report bugs
> in the http://bugs.sugarlabs.org tracker. How do we reconcile these
> two trackers?

Indeed, that's a problem.  Talk to Bernie and other Sugar Labs people
about their tracker.  They may have some ideas.

The OLPC trac exists for the engineering process covering hardware,
firmware, and operating system build to the point of deployment.  We
don't appear to be handling problem reports from deployments, or just
that the linkage is not clear.  We can handle Sugar or activity problem
reports in OLPC trac, and we do so when OLPC has to focus on such a
problem, but eventually they are forwarded on to Sugar Labs in some
form.

You face the same difficulty in reconciling the two trackers as everyone
else.

-- 
James Cameron
http://quozl.linux.org.au/