[OLPC-Games] vision processing / juggling

[Nirav Patel]

I think they want to wait on GSoC to finish before merging back into
the trunk.  In any case, it's going to release with pygame 1.9, which
is the next one after 1.8.1 comes out in a few days.

Nirav

On Tue, Jul 15, 2008 at 5:19 AM, Tim Wintle <tim.wintle at teamrubber.com> wrote:
> Nariv - this looks great - any idea when the camera module may get
> merged back into the pygame trunk?
>
> Tim Wintle
>
> On Mon, 2008-07-14 at 08:21 +0900, Nirav Patel wrote:
>> > Nirav Patel's GSoC project is vision processing for the XO (I believe
>> > using a subset of the OpenCV library)... and rather good at colored
>> > object / face recognition. (Eric, try the
>> > http://wiki.laptop.org/go/Colors! activity on an XO when you get a
>> > chance... it can use the green power cord (or really any light in a
>> > dark room) as drawing input)
>>
>> The original plan was to use OpenCV, but I switched to adding vision
>> stuff to pygame instead.
>>
>> > I was thinking ... a learn to juggle activity may make use of this
>> > processing to track progress and give tips (practice 1/2 balls again!,
>> > etc). Greg -- you've done some vision processing and you are a
>> > juggling guru... you might want to consider applying for a laptop at
>> > http://projectdb.olpc.at/ to play with this sort of thing. :)
>> >
>> > I wonder what best color / pattern could go on the balls to be easy to
>> > track... possibly to track spin? :)
>>
>> Any uniformly and brightly colored objects would work well.  For
>> convenience of coding, its probably best to use three different
>> colored balls.  Tracking spin would involve optical flow, which I do
>> plan on writing, but I'm not sure its feasible to run realtime on the
>> XO.  There is basically a two step process for color based object
>> tracking in the vision stuff I'm writing.
>>
>> The first step is finding the color of the object.  This involves the
>> user holding up an object so it fills a Rect being overlaid on images
>> being captured by the camera.  You would want to set up the Camera
>> object so that it captures images in HSV with the following:
>> camera.Camera("/dev/video0", size, "HSV")  It would then involve
>> looping something like the following until the user hits a key, where
>> cam is the Camera object, display is the display surface, and ccolor
>> is the average color you want:
>>
>>        snapshot = cam.get_image(snapshot)
>>        display.blit(snapshot, (0,0))
>>        crect = pygame.draw.rect(self.display, (255,0,0), (300,220,40,40), 4)
>>        ccolor = pygame.transform.average_color(snapshot, crect)
>>        pygame.display.flip()
>>
>> The second step is tracking the object.  It involves looping through
>> the following, where thresh is a surface the size and depth of
>> snapshot, cent[0] is the (x,y) coordinate of the object, and cent[1]
>> is the size in pixels:
>>
>>        snapshot = cam.get_image(snapshot)
>>        pygame.transform.threshold(thresh, snapshot, ccolor,
>> (20,25,25), (0,0,0), True)
>>        mask = pygame.mask.from_surface(thresh)
>>        cc = mask.connected_component()
>>        cent = cc.centroid()
>>
>> Note that if you're running this on the XO, you're going to want to
>> capture images as "RGB" at (320, 240), pygame.transform.scale them
>> down to (160, 120), convert the colorspace to "HSV", and then do all
>> the thresholding and mask stuff.  Note also that Camera.get_image()
>> returns 24 bit Surfaces, while the XO displays 16 bit surfaces.  So
>> blitting a Surface captured by the camera to the display involves a
>> strangely cpu expensive operation that makes it impossible to capture
>> and blit 640x480 images at 30 fps.  This may be something I can find a
>> workaround for, but for now, stick with 320x240 or smaller.
>>
>> There are a few example scripts in
>> http://git.n0r.org/?p=pygame-nrp;a=tree;f=examples/camera though none
>> of them are OLPC specific.
>>
>> Nirav
>>
>> > Brian
>> >
>> _______________________________________________
>> Games mailing list
>> Games at lists.laptop.org
>> http://lists.laptop.org/listinfo/games
>
> _______________________________________________
> Games mailing list
> Games at lists.laptop.org
> http://lists.laptop.org/listinfo/games
>