[Testing] QA team meeting notes - the summary

[Joseph A. Feinstein]

We held a QA team meeting on 11/5/08. Here is the summary.


Why AP tests?

Q: Why are we spending so much time running AP tests when the subject 
of APs for our international deployments is up to government bids, 
and not at our recommendation?

A: So that we have system level test results, and also so that we can 
make recommendations that the deployments can choose whether or not to take.

Comments:

- We should start writing down test cases for these situations so 
that those deployments can do their own testing for AP selection.
- So we know what test was for, how it was run, and why it was created.
- So we can repeat the test in the future.
- So we can log results and bug reports.
- For AP testing, it sounds like a great idea for us to be able to 
provide test cases for any interested deployments so they can run the 
tests themselves to ensure the AP they are considering will work for 
them. Testing at 1CC is done so we can make recommendations and we 
can say we tested with x, y, and z access points.


Backup/Restore testing

This testing involves a 24-hour wait; trying to reproduce the bugs. 
Still in the middle of these tests.

Comment: Backup testing should no longer involve a 24 hour wait.


Scripts

We have now a working "Deregister from school server" script that 
does the following:

- Removes XS line from config file.
- Deletes the network.cfg file.
- Sets the .xsession login.
- Sets the ds backup logging.
- Copies itself into /usr/bin so that it'll be permanently on your 
XO, and you can run it without the USB stick.

We will make a wiki section for these scripts.


Wiki reconstruction

Don't start it yet; collect ideas, but don't implement. Make sure we 
have an archive when the new architecture gets implemented.


Fedora on XO

It would be fine with one or two people joining the fedora-olpc test 
group and formally contributing a few hours/week on this testing as 
the olpc representative(s).


Automation

VNC: has been set up and running, but here are some issues:

- No frame control.
- Some Activities don't behave the same way when you start them from 
a remote machine vs. locally.

Ssh scripts: Let's first simplify and focus on running just the ps 
command and getting results back from it.

Tinderbox: has been on the back burner. Tinderbox and the ssh-scripts 
will let us run basically the same commands, but the tradeoff is that 
Tinderbox takes a long time to setup on an individual XO because you 
have to open it up and plug a special device into the motherboard, 
whereas the ssh-scripts can quickly be run on anything we have an IP 
for. However, Tinderbox doesn't rely on good connectivity to work.

Let's put together a small laptop testbed (1 or 2 to start) with one 
server that can run the tinderbox scripts. We can experiment on that 
bed just as we are experimenting with VNC, Monitor, etc. We need to 
explore uses of these tools to really understand the benefits.


Community test

We should try to get more volunteers involved. One of the outstanding 
questions from the test community is how to prioritize the Activities 
for testing.

Different opinions:

- Volunteers should test activities we don't test at 1cc. We test the 
behavior of Activities in a particular system situation.
- Volunteers should focus on testing activities that OLPC directly 
works on because we consider it core functionality, like Browse and 
Write, because we have more direct contact with the developers that 
make them. Browse, Write, Record, and Chat.
- Connect to jabber.laptop.org [so you can test Activity behavior 
independent of the mesh network] because it tests both collaboration 
and the XS.
  - People with one laptop can create test cases and help write up 
the 'documentation' of how an activity works.


Large testbed testing with only AP (no school server) connectivity

We're trying to see how many machines we can get up; have just 
started this, running into issues but finding ways to get past them. 
Side note: today we experienced a 2 hr email delay because our email 
server was overrun by kids in Uruguay accessing our wiki. Email 
processing and OLPC wiki should run on different machines.

New OFW firmware testing: first finish smoke tests, then move to a 
different testbed.