[Testing] QA team meeting notes - the summary

[Gary C Martin]

Hi Mel,

On 29 Oct 2008, at 20:11, Mel Chua wrote:

> This is awesome, Gary. I've created http://wiki.laptop.org/go/Scripts_for_testing_multiple_XOs 
>  for this project - it's definitely something we could use a hand on.
>
>>
>> Do you have some specific terminal commands you want to run? I'm  
>> guessing things like ps aux, free, ifconfig, nm-tool, avahi-browse - 
>> t _presence._tcp. I could knock up a bash script and test it here  
>> on the three XOs if you have a list of what you're after collecting.
> Here's what I'd love to have: http://wiki.laptop.org/go/Scripts_for_testing_multiple_XOs#Ideal_use_scenario
>
> Think you can do it? (Suggestions on better designs and ways of  
> interacting with remote XOs totally welcome - this is just the best  
> way I can think of for how I'd like to do it for our giant QA  
> testbeds here.)
>
> Thanks!

Could you give me a little more of an idea for what you're trying to  
achieve here. So far what I have read indicates that you want to, "run  
arbitrary batch file from master, on every XO in a given ip list, and  
store output back on master for later number crunching/data-mining."

Two fairly different needs could be going on here, I'm not sure which  
you are after :-)

1). Want to run some admin script on every XO. Perhaps you want to  
wipe the datastore and reboot, perhaps you want to trigger suspend and  
use administration ping to wake them up again repeatedly, etc.

2). Want to collect a set of data from every XO and look for anomalies/ 
failures. This could be lists of buddies each is currently seeing, or  
AP each is seeing, cpu loads, free memory, dropped packets, packet  
collisions, etc.

If option 1 is the primary use case, there is unfortunately a fairly  
lousy behaviour from a large number of the olpc and sugar custom  
commands where they require all kind's of X and custom environment  
variables set (as if you were working on the actual Terminal from  
Sugar in an X environment). Example you can't just use sugar-launch,  
or sugar-install-bundle without some magic environment stuff. So I  
couldn't guarantee just any script you might need to use would work  
without first resolving environment variable issues – I'll need to  
have a dig and see if there is a reasonable fix for this 'flotsam and  
jetsam' of Sugar custom environment stuff. If you just need to run  
standard bash scripts/tools, this problem goes away.

If option 2 is the primary use case, it would be a good idea to pick a  
range of useful data measurements and start with a simple clean data  
file. Something easily imported, displayed, graphed. You could see  
option 2 as a specific case of option 1, where you write a nice batch  
that collects and cleanly reformats all the spurious outputs you get  
from the various tools. If it is primarily data collection you're  
after, I would probably be better for me to build a script that  
reliably collects the needed measurements and stores it cleanly.

Regards,
--Gary