[Testing] Post-meeting conversation: Test results reporting

[Michael Stone]

Situational awareness is key to the ability to achieve results. Since
Mel seemed to respond positively to some of the remarks I made about the
situation of software development at OLPC, I thought that others might
benefit if I developed them in slightly more detail. Here's a first
pass, along with a restatement of the contents of Mel's email with nicer
formatting.

Michael

-----------------------------------------------------------------------

"Why are testers and test results valuable?"

My answer, from my perspective as someone who writes software and as the
8.2.0 release manager, goes something like this:

  1) An excited, vibrant test community helps motivate me to do my best
     work because my friends in testing deserve nothing less. 
  
  2) In spite of (1), I regularly make mistakes or exercise poor
     judgement. When I do, testers help protect me from the consequences
     by helping to discover, understand, fix, confirm the fix, integrate,
     confirm the integration, and finally, establish better procedures,
     thereby reducing the damage caused by the mistake.
     
  3) The broad application of (2) really means that it is testers' good
     judgement and perserverance which help most to remove inadequate
     work from the release queue before it hurts lots of people.

"So how to testers do all these wonderful things?"

Unfortunately, I don't have a great answer to this for all sorts of
reasons, so a poor one will have to do:

   a) search for the intersection of what you can do and what is needed.
       -- when necessary, invent new needs or to develop new skills so
          that you can better fulfill (a). seek advice when helpful.

   b) find the people who benefit or suffer from your work and figure out
      both how to talk to them in words they understand and what
      motivates them.

   c) help to bring out the best in the people around you and also find
      more people to do (a), (b), and (c).
   

Good Things So Far (and why they're good)
-----------------------------------------

Welly testers wrote up a spreadsheet for smoke testing <here>[1] (url 
needed). We understand that it works really well for them because:

  1) it helps them write up more uniform results. this makes life easier
     for testers, developers, and other stakeholders looking at the
     results later.

  2) it helps new testers get started, by prompting them on what to look
     for. This in turn lowers the cost of getting a new tester started,
     making it more likely that experienced testers will spend time doing
     it.

  3) it makes it easier for them to see and aggregate results at the end,
     since new testers tend to be more familiar with spreadsheets than
     semantic mediawiki.

  4) it helps new testers get started because they can just write down
     their thoughts without having to learn a complicated system to put
     their results into at first.


The Big Question
----------------

(which cjl thinks is inappropriate but which mstone thinks is motivated
by his criterion for success):

   "Is this spreadsheet-based approach a good way to communicate test
   results to the people who need to receive them?"

<<<< Your answer here >>>>



Test Results: a skit by Mel and Chris
-------------------------------------

* testers submit test reports, with little enforced structure

* the Community Testing group meets, looks it over, does triage and
   works out what's a legitimate bug

* the Community Testing group files bugs as necessary, taking note of
   the individual tester's willingness to take part in communication, and
   CC'ing them on the bug if they want to be.

-- cjb: (Hm.  This model of having testers be hands-off after they write
         a report is pretty much in conflict with our rationale.  What's
         up with that.)
-- mel: I'd phrase this as "There's a workflow within the test
         community," just as you have a workflow within the development
         community (as m_stone's trac tickets workflow wiki page so
         elegantly illustrates.) People can choose to participate in as
         little or as much of that workflow as they choose; I could
         imagine an awesome test volunteer deciding that they just want
         to triage, for example.  What we are talking about here is
         separating the "Actually run and test the software" part from
         the "and report it to developers in the most useful format/form
         for them" part. Just like the person who makes the patch doesn't
         have to be the one that includes it in a build, etc.


Utopia according to Chris & Mel
-------------------------------

cjb's developer idea of what a tester's utopia might be:

Testers find a process that is unburdensome and helpful. They know that
they can be as involved as they want to in the work done on their bugs.
They're asked, if they want to be, things like "Do you think this change
would solve your problem?". Their work is documented, shared,
appreciated, and credited. Someone running an OLPC/Sugar deployment can
look up what tests have been performed on a component; this information
should even be part of the release notes for software. ...

mchua's tester idea of what a developer's utopia might be:

Developers find a process that is unburdensome and helpful - and usually
invisible in the background. By default, they're notified only of bugs
(via well-written, reproducible bug reports) on the components that they
care about, and this notification uses the communiation channel(s) they
prefer (trac, email, IRC, etc.) The process by which the bug report was
created is transparent, and they can go back to the testers who carried
the bug through that process and open up a dialogue with them, asking
questions and going back and forth as they fix the bug and then verify
the fix. Their work is documented, shared, appreciated, and credited,
and they even get a follow-up from someone (perhaps a tester) after
their fix has later been pushed to a release and deployed, saying
"Thanks - and here's the difference you made to the users you were doing
this for!"

Addendum: Criteria for success
------------------------------

mstone's proposal:
    We need to be careful to pay attention to results, not just efforts.
    For this reason, we should take care to judge collaboration protocols
    we invent according to efficiently they inform and motivate the total
    set of people who need to work together to solve the Big Problem, not
    according to how well they solve any individual small problem.

mchua agrees, though she's not yet sure how to measure whether something is
more informing or motivating than something else, in this context.


Addendum:  Rationale:
---------------------

<cjb> it would be pretty sweet to have a section in the release notes
        called "What The Testers Found"
<cjb> with a description of which groups tested which parts of the
        release, and what they thought and stuff
<cjb> that would be pretty grassroots, right there


Things potentially worth knowing about current attitudes toward bugs
--------------------------------------------------------------------

If information from these results were subsequently filed as
well-written bugs, 
     
   a) interested developers would be happy because individual bugs are
      unlikely to be fixed unless well-filed bug reports are filed. 

   b) uninterested developers be happy because they wouldn't be
      distracted.
       
   c) people with established daily workflows based on email and
      ticket-based meetings would be happy because they can continue
      using their workflow
    
   d) release management people would also be happy because bugs are
      the preferred way to keep track of the status of the project
      and its resource allocations.