[OLPC-Games] vision processing / juggling

[Nirav Patel]

> Nirav Patel's GSoC project is vision processing for the XO (I believe
> using a subset of the OpenCV library)... and rather good at colored
> object / face recognition. (Eric, try the
> http://wiki.laptop.org/go/Colors! activity on an XO when you get a
> chance... it can use the green power cord (or really any light in a
> dark room) as drawing input)

The original plan was to use OpenCV, but I switched to adding vision
stuff to pygame instead.

> I was thinking ... a learn to juggle activity may make use of this
> processing to track progress and give tips (practice 1/2 balls again!,
> etc). Greg -- you've done some vision processing and you are a
> juggling guru... you might want to consider applying for a laptop at
> http://projectdb.olpc.at/ to play with this sort of thing. :)
>
> I wonder what best color / pattern could go on the balls to be easy to
> track... possibly to track spin? :)

Any uniformly and brightly colored objects would work well.  For
convenience of coding, its probably best to use three different
colored balls.  Tracking spin would involve optical flow, which I do
plan on writing, but I'm not sure its feasible to run realtime on the
XO.  There is basically a two step process for color based object
tracking in the vision stuff I'm writing.

The first step is finding the color of the object.  This involves the
user holding up an object so it fills a Rect being overlaid on images
being captured by the camera.  You would want to set up the Camera
object so that it captures images in HSV with the following:
camera.Camera("/dev/video0", size, "HSV")  It would then involve
looping something like the following until the user hits a key, where
cam is the Camera object, display is the display surface, and ccolor
is the average color you want:

       snapshot = cam.get_image(snapshot)
       display.blit(snapshot, (0,0))
       crect = pygame.draw.rect(self.display, (255,0,0), (300,220,40,40), 4)
       ccolor = pygame.transform.average_color(snapshot, crect)
       pygame.display.flip()

The second step is tracking the object.  It involves looping through
the following, where thresh is a surface the size and depth of
snapshot, cent[0] is the (x,y) coordinate of the object, and cent[1]
is the size in pixels:

       snapshot = cam.get_image(snapshot)
       pygame.transform.threshold(thresh, snapshot, ccolor,
(20,25,25), (0,0,0), True)
       mask = pygame.mask.from_surface(thresh)
       cc = mask.connected_component()
       cent = cc.centroid()

Note that if you're running this on the XO, you're going to want to
capture images as "RGB" at (320, 240), pygame.transform.scale them
down to (160, 120), convert the colorspace to "HSV", and then do all
the thresholding and mask stuff.  Note also that Camera.get_image()
returns 24 bit Surfaces, while the XO displays 16 bit surfaces.  So
blitting a Surface captured by the camera to the display involves a
strangely cpu expensive operation that makes it impossible to capture
and blit 640x480 images at 30 fps.  This may be something I can find a
workaround for, but for now, stick with 320x240 or smaller.

There are a few example scripts in
http://git.n0r.org/?p=pygame-nrp;a=tree;f=examples/camera though none
of them are OLPC specific.

Nirav

> Brian
>