[OLPC-Games] vision processing / juggling

[Tim Wintle]

Nariv - this looks great - any idea when the camera module may get
merged back into the pygame trunk?

Tim Wintle

On Mon, 2008-07-14 at 08:21 +0900, Nirav Patel wrote: 
> > Nirav Patel's GSoC project is vision processing for the XO (I believe
> > using a subset of the OpenCV library)... and rather good at colored
> > object / face recognition. (Eric, try the
> > http://wiki.laptop.org/go/Colors! activity on an XO when you get a
> > chance... it can use the green power cord (or really any light in a
> > dark room) as drawing input)
> 
> The original plan was to use OpenCV, but I switched to adding vision
> stuff to pygame instead.
> 
> > I was thinking ... a learn to juggle activity may make use of this
> > processing to track progress and give tips (practice 1/2 balls again!,
> > etc). Greg -- you've done some vision processing and you are a
> > juggling guru... you might want to consider applying for a laptop at
> > http://projectdb.olpc.at/ to play with this sort of thing. :)
> >
> > I wonder what best color / pattern could go on the balls to be easy to
> > track... possibly to track spin? :)
> 
> Any uniformly and brightly colored objects would work well.  For
> convenience of coding, its probably best to use three different
> colored balls.  Tracking spin would involve optical flow, which I do
> plan on writing, but I'm not sure its feasible to run realtime on the
> XO.  There is basically a two step process for color based object
> tracking in the vision stuff I'm writing.
> 
> The first step is finding the color of the object.  This involves the
> user holding up an object so it fills a Rect being overlaid on images
> being captured by the camera.  You would want to set up the Camera
> object so that it captures images in HSV with the following:
> camera.Camera("/dev/video0", size, "HSV")  It would then involve
> looping something like the following until the user hits a key, where
> cam is the Camera object, display is the display surface, and ccolor
> is the average color you want:
> 
>        snapshot = cam.get_image(snapshot)
>        display.blit(snapshot, (0,0))
>        crect = pygame.draw.rect(self.display, (255,0,0), (300,220,40,40), 4)
>        ccolor = pygame.transform.average_color(snapshot, crect)
>        pygame.display.flip()
> 
> The second step is tracking the object.  It involves looping through
> the following, where thresh is a surface the size and depth of
> snapshot, cent[0] is the (x,y) coordinate of the object, and cent[1]
> is the size in pixels:
> 
>        snapshot = cam.get_image(snapshot)
>        pygame.transform.threshold(thresh, snapshot, ccolor,
> (20,25,25), (0,0,0), True)
>        mask = pygame.mask.from_surface(thresh)
>        cc = mask.connected_component()
>        cent = cc.centroid()
> 
> Note that if you're running this on the XO, you're going to want to
> capture images as "RGB" at (320, 240), pygame.transform.scale them
> down to (160, 120), convert the colorspace to "HSV", and then do all
> the thresholding and mask stuff.  Note also that Camera.get_image()
> returns 24 bit Surfaces, while the XO displays 16 bit surfaces.  So
> blitting a Surface captured by the camera to the display involves a
> strangely cpu expensive operation that makes it impossible to capture
> and blit 640x480 images at 30 fps.  This may be something I can find a
> workaround for, but for now, stick with 320x240 or smaller.
> 
> There are a few example scripts in
> http://git.n0r.org/?p=pygame-nrp;a=tree;f=examples/camera though none
> of them are OLPC specific.
> 
> Nirav
> 
> > Brian
> >
> _______________________________________________
> Games mailing list
> Games at lists.laptop.org
> http://lists.laptop.org/listinfo/games