#8106 - a testing case study

[Mel Chua]

http://dev.laptop.org/ticket/8106 makes an interesting, relatively 
easy-to-follow (even for nontechnical folks) case study on how testing 
works right now, since improving QA processes for the next release cycle 
has been a topic of discussion lately.

Note that the snags we hit were minor and easily fixable, but took a 
while to resolve - how can we fix this for next round?

Here goes. Mel's Ideal scenario (vs what happened):

1. There'd be a test case attached to the ticket saying "Do this! If it 
passes, you've proved the feature works."

(Reality: No test case. No problem - I wrote one up. Getting good 
testcases to happen is something the QA team should do in any case.)

2. But wait. To write the test case, we needed to find an out of date 
content bundle that we could upgrade. Ideally, there would be a list of 
available content bundles, you could find different versions of each 
content bundle, and install an old one.

(Reality: I hunted around in vain; SJ, bless him, finally stepped in and 
gave me direct URLs to 2 bundle files, simultaneously noting that there 
was not, in fact, a good way to find that kind of download/history 
information yet.)

3. Ok, great - we've installed the old version of the library bundle, we 
run the software updater, and we see a clear difference in behav

(Reality: the fault was mine for making a test case that didn't include 
the last verification steps for "is this bug fixed?" I'd expected some 
sort of visible indication on the screen saying "your content bundles 
are updating." There wasn't a message like that, but after a few minutes 
of confusion, I realized I should check the library.info file manually. 
This indicated that the bundle had indeed been upgraded. I filed 
http://dev.laptop.org/ticket/8805 suggesting that we make this visible 
to users next time.)

4. Close bug.

(Reality: Close bug. Yay!)

-- 

Mel Chua
QA/Support Engineer
mel at laptop.org