[olpc-nz] [Testing] Improving the reporting of test results

Gary Martin garycmartin at googlemail.com
Wed Dec 22 00:41:29 EST 2010


On 22 Dec 2010, at 04:51, Sridhar Dhanapalan <sridhar at laptop.org.au> wrote:

> We at OLPC Australia have been the recipient of some really good
> testing feedback (especially from NZ) on the lists. However, I also
> perceive a major weakness in the process. My central question is, how
> do we ensure that the feedback actually gets to the relevant people in
> a useful way?
> 
> The executive summary is that all bugs/issues should be reported in an
> issues tracking system (redmine, trac, bugzilla, etc.), so that the
> developers can easily tend to and manage them. I'll explain...
> 
> The basic develop-test-report cycle goes like this:
> 
>  1. software/hardware is developed
>  2. testers give it a spin
>  3. testers note their results
>  4. developer easily sees the feedback
>  5. developer can easily manage the feedback
>  6. based on the feedback, GOTO 1
> 
> What I see is that steps 1-3 are being handled quite well.
> 
> There appears to be a big hole between steps 3 and 4. This is because
> the testing feedback is only posted as prose on the mailing lists.
> There's nothing to be sure that the relevant developers are seeing
> those messages. Given the sheer volume of list messages, they probably
> aren't.

Personally I find the posts very informative and useful, I try to reply and open tickets if needed for activities I help maintain (or if I've seen an annoying issue crop up before). I actually like reading the email prose, as often it gives much more context and feeling than a dry 'your activity is defective' ticket. The emails also often drift into the surrounding attempted use case, and suggest other design niggles, frustrations, and misunderstandings that might otherwise be missed. Of course tickets are great and appreciated when you have a specific feature request, or reproducible bug, but the email stage first seems like a good low barrier to actually getting more reports.

Too be brutally honest, what we are short of is folks with the time, ability and motivation to take concrete action, though having tickets is a great way for a maintainer to prioritise work when they do sit down to make a new release:

  5.5. developer adds feature / fixes bug and releases a new bundle ready for testing

Regards,
--Gary

> Even if the developer does see the feedback, how does (s)he manage it?
> This is what issues tracking systems are for. If an issue is properly
> reported, it can be properly managed along with the other tasks
> required for the project. Statuses, priorities and owners can be
> assigned. Now that it's in the system, it won't get lost. This is all
> neatly described at http://wiki.laptop.org/go/Reporting_bugs
> 
> If step 4 occurs but not step 5 (i.e. prosaic feedback is received,
> not input into an issues tracking system), it is up to the developer
> to turn the prose into a bug report. This is time consuming and
> detracts from the act of development (step 6).
> 
> In summary, I strongly urge testers to fully report their findings in
> the appropriate issues tracking system. In doing so, you make sure
> that the developers see your findings (thus making your testing
> worthwhile) and can easily act upon them.
> 
> Examples of tracking systems to submit to:
> 
>  OLPC: http://dev.laptop.org/
>  Sugar Labs: http://bugs.sugarlabs.org/
>  OLPC Australia: http://dev.laptop.org.au/
> 
> 
> Cheers,
> Sridhar
> 
> 
> Sridhar Dhanapalan
> Technical Manager
> One Laptop per Child Australia
> M: +61 425 239 701
> E: sridhar at laptop.org.au
> A: G.P.O. Box 731
>      Sydney, NSW 2001
> W: www.laptop.org.au
> _______________________________________________
> Testing mailing list
> Testing at lists.laptop.org
> http://lists.laptop.org/listinfo/testing


More information about the olpc-nz mailing list