[Testing] [sugar] Automated testing, OLPC, code+screencasts.

[Benjamin M. Schwartz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Titus Brown wrote:
| Other than that, I'm interested in figuring out how to encourage the
| OLPC project to build some automated tests without doing it all myself
| :).  Thoughts welcome.
|
| Basically I think it's incredibly important for the OLPC to invest in
| automated tests.

There are many people in and around OLPC who feel the same.  There are
also a few who are exceptionally familiar with Tinderbox, and for a time
we had a tinderbox running tests on every build, hooked up to an actual XO
to maximize realism.  The closest I'm aware of to actual GUI testing was
ensuring that Sugar started successfully.  See
http://wiki.laptop.org/go/Tinderbox_Testing .

| As you can guess from the "death spiral" post, I'm not terribly
| interested in *debating* the usefulness of automated tests, and I don't
| buy into the notion that GUI automation is impossible, either.  I would
| very much like to try to introduce systematic and robust GUI testing
| into the OLPC and I will be working towards that end as time and
| resources permit.  Constructive comments and genuine interest are
| welcome!

You will get a warmer welcome if you take a more positive tone.

Personally, I am skeptical, for several reasons:
0. We already know what the bugs are.  The problem is not new bugs, and
it's not regressions.  The problem is bugs that we've known about for a
long time that are just not easy to fix.
1. Regressions are rare, and they are not the main problem.  At least 90%
of OLPC's problems fall into the category of unimplemented functionality,
not really bugs at all.  Recently, a number of features have been removed
because we finally did enough testing to discover underlying bugs, but
these were not regressions.
2. Many, and perhaps most, of OLPC's remaining difficult bugs are related
to the network.  They are most commonly related to the closed wireless
firmware, which is buggy and lacks key features regarding mesh routing and
multicast.
3. Almost all of OLPC's major bugs are Heisenbugs.  They often don't
appear at all with only one laptop, and appear rarely until one has 12 or
more laptops sharing a wireless mesh.

One way to prove me wrong would be to filter the ~6800 bugs in Trac into
"Regression", "Non-regression", and "Other".  I think you will find that
there are very few regressions.  Another perfectly good way to prove me
wrong is to build a complete testing suite and start spotting regressions,
but that's more difficult.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH6vYXUJT6e6HFtqQRAv5/AKCgG/nTudD44HmCVkXSgPy76Ds+KgCfTqer
lncEU/e0rFBDXiXkG8EW1UY=
=zpnk
-----END PGP SIGNATURE-----