[OLPC-Games] Computer vision on the XO: what it is. what should it be?

[Tim Wintle]

I know this isn't the fun side of this (or at least I wouldn't find it
the fun bit), and perhaps OT for OLPC games - but ensuring that the
webcam works on as many platforms and webcams as possible would be very
high up for me.

I wouldn't bother with object recognition - it's too general and I
haven't seen systems that work well enough for anyone that doesn't know
how hard it is to write to be truly impressed.

On the other hand, face detection would be *really* useful - e.g. for
taking a shot of someone's faces to add to high score tables etc. There
was a very good C library around to do it somewhere online if you can
find it (not sure of the licence) - I got around 98% accuracy with it.

Other possible ideas - with ideas for applications if someone cares to
write them (I don't have time):

 * OCR - So teachers / pupils can take a picture of the blackboard and
have it converted to text.

 * Barcode reading - Did anyone ever have the "barcode battler" handheld
game? It used the value of a barcode to create a tamagotchi-style
character you could battle your friends with - so any barcode you could
find could be the best monster to beat your friends with.



Obviously face recognition would also be great, but I'm guessing you'd
have to use eigenface-style methods which aren't that good on unseen
data in my experience - although they're very quick to run, so they
could work. I don't know if "tensor faces" was patented when it was
released (I think around 2006), but that seems good.


On Wed, 2008-07-23 at 11:43 +0900, Nirav Patel wrote:
> I'm writing computer vision functions for Pygame (available at
> http://git.n0r.org/?p=pygame-nrp;a=summary ), and I've gotten to the
> point where I very much need community input on where to go next.
> Basically, I would like to know what you want to be able to do with
> the camera on the XO, whether its related to gaming, input,
> accessibility, education, or anything else.
> 
> What it can currently do, in overly simplistic terms:
> 
> 1. Capture images:  This is the basis for everything else, but it can
> be useful on its own too.  For example, letting someone take a picture
> of themselves, let them select their face in the picture, and then
> crop that and use it as a character in a game.
> 
> 2. Get the average color:  This is useful for picking values to
> threshold, but you can also switch to YUV or HSV colorspace and find
> the average brightness of the area.
> 
> 3. Threshold images:  Thresholding is pretty flexible.  You can have
> it select everything within a threshold of a color or everything
> outside the threshold of a color.  You can also threshold between two
> images.  This lets you get a "green screen" effect, so you can have a
> person being displayed realtime over a virtual background.
> 
> 4. Track an object:  After thresholding, you can turn the remaining
> object into a bitmask and get various properties about it like its
> bounding box, centroid, size in pixels, and angle with respect to the
> x-axis.  You can also test collisions between the mask and masks of
> virtual objects.
> 
> They may seem pretty basic and simple, but you can do a lot of things
> by combining them.  For example:
> 
> 1.  Drawing with a real life object:  Have the user pick up an object
> and hold it up so it fills a box being displayed on screen and hit a
> button.  Save the average color within the box.  Threshold out just
> that color, turn it into a bitmask, get the largest connected
> component, and find the centroid of it.  Use that centroid as the
> coordinates for the on screen paint brush, perhaps also using the
> saved average color.  The user now has the illusion of using the
> object in hand as a paint brush.
> 
> 2.  Play pong with your hand:  Have the user step out of the field of
> view of the camera, save the image.  Now threshold between the saved
> image and the images currently being captured.  This results in just
> showing the differences between the two (like the user, who has now
> stepped back into the field of view).  Turn the image into a bitmask
> and check it for collisions with the bitmask of the ball in pong.
> Actually, now that I'm thinking about it, I'll probably write this
> game when I get home.
> 
> Other examples: http://eclecti.cc/olpc
> 
> What it doesn't do yet, but could depending on if there is interest:
> 
> 1. Generic motion detection:  You can do stuff like thresholding
> between a background and current images, or tracking whether blobs of
> colors have moved, but neither is a great way of detecting total
> motion in an image.  There are many optical flow algorithms, but
> finding one that'll run realtime on an XO will be tricky.  The other
> issue is how to present motion to the developer.  Perhaps have the
> function request two images and a list of points on the first image,
> and return a list of where it guesses the points are now on the second
> image.
> 
> 2. Object recognition:  You could guess based on the size or color of
> an object, but there isn't really a way of detecting if what you're
> holding up is a lime or a pear.  There are a lot of really
> computationally heavy ways of doing object recognition, but there are
> also some lightweight ways.  The one I was considering was using image
> moments (which I am currently doing to find the centroid and angle of
> an object), to get basic parameters like the eccentricity and skewness
> of an object.  There is also the Hu set of invariant moments that will
> give more information about an object, though not really in a
> human-friendly form.  Thus, while I could fairly easily write
> functions that would drop a dozen of these numbers, I'm not sure
> anyone would be able to make use of them in an Activity.
> 
> So far, I've just been writing functions based on usage cases I could
> think of in games.  There are probably many that I just haven't
> thought of for which the current functions are inadequate.  So, any
> ideas would be appreciated.
> 
> Nirav
> _______________________________________________
> Games mailing list
> Games at lists.laptop.org
> http://lists.laptop.org/listinfo/games
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.laptop.org/pipermail/games/attachments/20080723/cf4b763d/attachment-0001.pgp