[Testing] Community-based testing of OLPC activities

[Greg DeKoenigsberg]

On Wed, 7 Nov 2007, Kim Quirk wrote:

> Hi Greg,
> Thanks for putting this together. I think the matrix of test plans is
> a good idea. We had tried to create something like this at
> http://wiki.laptop.org/go/User_Stories_Testing site. Yours is a more
> friendly and probably easier to use, so it would be great if you can
> incorporate your look/style with the test plans and info that are
> there.
>
> My biggest issue with a wiki based test case management system is that
> you can't easily track results. Test cases are rarely run once -- but
> need to be run many times after pretty much any code changes that
> might affect it. If there is some way to hook this 'result' page into
> a test case management database, then we would get the best of both
> worlds. Easy update, easy run and good reporting on a wiki page.

Yep.  I regard this as a stopgap.  It's a bit of chicken-and-egg; it's 
easier to prove the need for infrastructure when you've generated demand 
for it, in my experience.

> The kind of reporting we need is: date/timestamp for each run; tester;
> pass/fail results; link to bugs
> Do you think that can be done in the wiki?

We can certainly try.  More overhead == harder, but so long as the 
template is simple, it should be manageable.  Mostly, what you need is 
someone who's committed to improving the process over time.

I'm willing to be that guy for a while, since it dovetails with the 
"community action" theme.

> Another idea that people have had -- but no time to implement -- is a
> Test Activity. This activity could be loaded onto the laptop or
> simulator and run from there to provide you with all the test cases.
> You pick a test case, run it, add your pass/fail, comments and or bug
> tracking numbers and when you hit 'submit' it sends the results back
> to our testcase mgmt server.
>
> To me this would be ideal... but it seems very difficult to pull
> together all the pieces. Someone is working on the log viewer,
> gatherer (it is important to have the logs whenever there is a
> problem); others have thought about a Test Activity; and some have
> looked into a test case mgmt database, but have not found something we
> can use.

We've got some stuff in redhatland for this -- I wonder if we could make 
it work for OLPC?  One of our QA studs here, who does this kind of test 
harness automation, volunteered some time.  I'll ask him if he has any 
advice.  And yes, it's hard to Do It Right without a significant 
investment of time and energy.

> Thanks for your thoughts and ideas as well. I have copied the
> 'testing' mailing list. A group of people interested in the testing
> problem. Can I add your email to this list?

I would be honored.  In the meantime, I'm going to try to merge our test 
ideas into a second revision, and I'll send it your way.  Probably be in a 
day or two.

--g

-- 
Greg DeKoenigsberg
Community Development Manager
Red Hat, Inc. :: 1-919-754-4255
"To whomsoever much hath been given...
...from him much shall be asked"