[Testing] So It Begins, Part 1: Community testing meeting logs, 2008-12-11

[Mel Chua]

Now with actual notes! (I'll say it again: BEST MEETING EVAR. You all 
*rock.*)

Next meeting: 
http://wiki.laptop.org/go/Community_testing_meetings/2008-12-18

Log: http://meeting.laptop.org/olpc-meeting.log.20081211_1800.html

Notes (same as this email's text): 
http://wiki.laptop.org/go/Community_testing_meetings/2008-12-11#Meeting_summary

First of all, it's entirely too quiet on this mailing list. C'mon, folks 
- ask questions, post notes, and talk here - don't let me monopolize the 
conversation. ;)

So, Activity testing: 2 weeks to go, and we're in good shape. We have 
got (1) stuff to test, (2) a way to test it, and (3) testing parties and 
sprints springing up on... heck, we might be able to hit every continent 
except for Antarctica (I have a couple friends who worked in Antarctica 
last year, though, so... *typetypetype* ok, I just sent an email. We 
shall see.) We've come a long way; figuring out what you're going to do 
is often the largest part of actually doing it. ("Stating a problem 
clearly is the first half of the solution to that problem.")

Second, we didn't really talk about this at the meeting, but Michael 
Stone is looking for someone to take over 
http://wiki.laptop.org/go/Friends_in_testing. We're asking the 
Fedora-on-XO testers to step up here since a lot of what we think we're 
going to be hunting for is bugs related to the rebase to Fedora 10 (F10, 
up from F9), but more eyeballs mean shallower bugs, so ping michael at 
laptop dot org if you're interested.

The exciting part for me was the last 20 mins when we talked about 
http://wiki.laptop.org/go/Community_testing_meetings/2008-12-11#Joyride.2F8.2.1_Testing:_The_Challenge. 
The short version is that we've been challenged to a friendly 
head-to-head competition with OLPC's internal QA team for testing 8.2.1 
(a small interim release, see http://wiki.laptop.org/go/Eco/8.2.1 and 
also look on trac for bugs with milestone: 8.2.1)

There are two goals to this: the first is, obviously, testing 8.2.1, but 
it is also an experiment to see the difference between internal QA 
testing and volunteer testing in terms of investment of internal OLPC 
resources + the return of that investment. More details in the log and 
at the very end of this email.

Finally, the Learning Stuff links of the week (got QA/test/related 
resources you'd like to share? Send them to this list!):
http://mako.cc/writing/funding_volunteers/funding_volunteers.html - Mako 
has an interesting perspective on paid vs volunteer developers in 
open-source, which might be relevant to our current 8.2.1 experiment.

A 40-page software testing primer, good for people new to QA who want to 
pick up terminology fast (it's written conversationally and written 
well): http://www.nickjenkins.net/prose/testingPrimer.pdf

A short 1-page intro to exploratory testing (the kind of Activity 
testing we are doing now): 
http://www.stickyminds.com/sitewide.asp?ObjectId=2255&ObjectType=COL&Function=edetail

That's all, folks - have a great week!

--Mel

PS: More notes on the 8.2.1 testing thing, below.

Rough metric: quality/time, where time = man-hours of internal QA time 
spent either facilitating community test, or running tests ourselves, 
and quality = number of test case runs completed satisfactorily, where 
"satisfactorily" is some bar that has yet to be defined. (Help making 
this experiment more well-defined is welcomed, but we have enough to do 
start and do a rough one, at least.)

What that means is that I'm going to be trying to be as hands-off about 
community 8.2.1 testing as possible, while (this is the hard part) still 
making sure you folks have everything you need. What *that* means is 
that I will *not* be spending time during 8.2.1 testing asking people 
what they need (as much as I can stop myself from doing so) and that you 
have to tell me. Better yet, as Michael said, tell each other, and help 
each other out.

The only thing you have to be synced up with me on is being very, very 
clear on what the metric and the bar we're trying to hit is, because as 
of now, we're racing. Talk with each other. Talk with developers. Do 
what you need to thrash me soundly (I'll be running as fast as I know 
how). ;)