[Testing] GUI testing (was: Automated testing, OLPC, etc.)

[Titus Brown]

On Thu, Mar 27, 2008 at 12:45:04AM -0700, Charles Merriam wrote:
-> Hello Titus,
-> 
-> Good to see you at PyCon.  Here's a random idea I think would be
-> worthwhile, if it could work well into the Sugar-gui or even the other
-> subsystems of Sugar.
-> 
-> Tooting my own horn today,
-> 
-> Charles Merriam
-> 
-> ===
-> http://www.charlesmerriam.com/blog/?p=106
-> Problem:  Testing GUIs tends to be hard.
-> 
-> Writing a reasonable test for GUIs usually involves contortions to
-> find the correct widget and values.  For example, a test might want to
-> confirm that "the font is now bold for the third text field, named
-> system_danger_level, in the hbox in the floating frame in the second
-> panel in the third tab bar in the dialog box".  Figuring out how to
-> tell if the test passed is usually difficult.
-> 
-> Solution:  Add a function in the GUI framework that returns a single
-> large data structure for the state of the GUI.  Use standard Python
-> programming to navigate it.

Hi Charles,

a few comments:

This is definitely one of two approaches I'd like to try.  (The other
approach is to try driving the GUI "in process" via a thread that
accepts requests via XML-RPC, but that could run afoul of GTK issues.)

Either way, I think that the right level of encapsulation would be to
provide a simple twill-like domain-specific language [0] with commands
like 'click' and 'select' that encapsulates likely user actions, and
then write a backend with the intelligence necessary to support
this.  This backend could be embedded in the app and manipulated via
RPC, or use the accessibility layer and/or take information directly
from Sugar and/or require a bit of instrumentation, or ...

In any case I agree with you that the best approach would be to try out
a few different simple schemes (perhaps on Calculator) and then see
which one is most robust and most likely to actually work in the long
run.

The nice thing with the twill/DSL approach is that down the road,
multiple GUIs could support the same up-front syntax.  Wouldn't that be
nice? ;)

Oh, and one last comment -- re figuring out how to tell if the test
passed, that's something that can be done by peekaboo and minimal
invasion of the code base under test.

cheers,
--titus

[0] twill example: http://twill.idyll.org/examples.html