[olpc-nz] [Testing] Improving the reporting of test results

Mikus Grinbergs mikus at bga.com
Wed Dec 22 11:39:19 EST 2010


Topic:  What should be reportable ?

A step missing from that 'basic develop-test-report cycle' list:

>   0. knowing what to expect

Speaking for myself, I do not have the time to read all the existing
wikis and tracking systems to discover what has been decided upon to be
provided (and equally importantly, what has been decided upon to be
omitted).  As a result, what I report is based on my own mental picture
of how the system to be tested ought to behave -- and my mental picture
can often be different from what the developers have in mind.

{In industry, specifications are often written in the form of "shall"
clauses.  [For example:  "when AC power is removed, the product shall
continue its normal operation on battery power alone, for at least two
hours".]  Then at the minimum, 'testing' needs to assure that each such
"shall" clause has been correctly addressed.}

----

I've seen tickets which describe behavior which the person doing the
testing did not expect - yet there were facilities available (e.g., the
Sugar Control Panel [a.k.a. 'My Settings'] with which the user himself
could have changed the behavior to what he preferred.  Presumably such a
ticket is still valid - because schoolchildren out in the field could
run into the same situation - but there is little a *developer* can do.

My own conclusion:  It is not enough for tracking systems to exist so as
to provide feedback to developers.  There ought to be tracking systems
in place to provide feedback (to management?; to tech writers?) as to
the *usability* of the product.

----

Then there is the question of "repeatability by developers".  [I stay
away from reporting Fedora bugs - to me it feels like excessive work to
gather all the "repeatability" information they expect.]  It often
interrupts what I am currently focused on to be asked to follow-up on
something I've reported before.  As a result I now self-censor myself -
and write tickets only for problems that can be "demonstrated".  I
figure that for "occasional" problems -- if enough people see them --
*they* will write up that ticket.


> If an issue is properly reported, it can be properly managed

What I've tried to express here is that "issues" in testing a product
often have loose ends.  As such, some people (including myself) may not
find it easy (as someone else might hope it was) to "properly report"
our experience.  [Even deciding "whether to report" might give pause.]


mikus



More information about the olpc-nz mailing list