[Sugar-devel] Sugar with a virtual (onscreen) keyboard

[Sayamindu Dasgupta]

On Fri, Jun 18, 2010 at 9:04 AM, Gary Martin <garycmartin at googlemail.com> wrote:
> Hi Sayamindu,
>
> On 17 Jun 2010, at 13:16, Sayamindu Dasgupta wrote:
>
>> [Apologies for the cross-posting]
>>
>> Hello,
>> Thanks to the pointers provided by Peter Robinson, I got the Meego
>> FVKBD (Free Virtual Keyboard)¹ running along with Sugar.
>> A problem with the current FVKBD is that it supports only one base
>> layout. Even "variants" of that layout (eg: CapsLock enabled, Symbols,
>> etc) are treated as "temporary", which means that you press the "Caps"
>> key, enter a capital letter, and immediately after that, it gets reset
>> back to the base layout (lower case qwerty).
>> I wanted something which would be similar to the existing physical
>> keyboards that we ship with the XO machines - with a dedicated key to
>> switch between different scripts in the same keyboard. I had to extend
>> the code of FVKBD to implement that, and with the modified FVKBD, I
>> have spun a live-cd ISO (based on the current SOAS). You can download
>> it from http://dev.laptop.org/~sayamindu/sugar-vkbd-test/sugar-vkbd-test.iso
>
> Wow, big thanks for launching into this. For anyone not sure how to try the iso, I'm on a Mac and just used Virtual Box to create a new empty Fedora VM, no HD, and just point to the iso as the boot CD. Started up just fine, keyboard is already open to type in your user name (of course this is all read only, any changes you make will be gone after a reboot).
>

Thanks for the feedback - this is really helpful :-)

> I'll try and spend some time in the next few days using it via iPad HW and send some feedback, just been playing via mouse so far today.
>
>> Apart from the modified FVKBD, I have added a default keyboard
>> definition file which is for English + Bengali, and I've also included
>> a sugar device-icon on the frame to control the appearance of the
>> keyboard.
>>
>> I realize that more needs to be done to support non Latin scripts, and
>> here are some of the issues I faced while converting the existing XKB
>> Bengali layout:
>>
>> * Many scripts do not have a concept of upper case/lower case - so we
>> need some other script specific way to divide the characters
>> * In the current XKB configurations, non-symbol characters from other
>> scripts are often placed in the position of what normally is symbols
>> for QWERTY keyboards
>> * Numerals pose an interesting problem, since in some places, native
>> numerals/digits are quickly being obsoleted, and latin numerals
>> (1,2,3..) are becoming the de-facto standard. In these cases, it may
>> make sense to provide only _one_ layout/state for numerals, and allow
>> users to input native numerals by hovering (touch + hold) on the
>> virtual key for the latin digit.
>>
>> Among the general issues, I'm not sure how to deal with the keyboard
>> taking up half of the screen real estate - it may be worthwhile to see
>> if we can have a "split screen" sort of configuration while the
>> keyboard is active.
>
> It didn't bother me too much, and this was in an 800x600 session, though ideally we would want the text insertion point to be visible above the keyboard (FWIW various iPad apps have different success in dealing with this, all of Apple's are fine, but it seems 3rd parties do need to do some work on the app side to keep this behaviour working at all times).
>

Transparency is something which comes to mind. Another possibility
might be to make the keyboard move up to the top half of the screen
after a certain point - but that may be too annoying.

>> Thoughts, feedback, etc would be appreciated :-).
>
> Yes, lot's of interesting items to cover :-) I'll try to start to put together a list. Some quick item that struck me right away:
>
> - the Meego keyboard design is clearly for casual typing/text entry, no way of typing commands or many symbols needed for basic programming work – diving into terminal to use vi, or worse emacs, is pretty much a dead end (unless ctrl and alt keys are hidden somewhere I couldn't find). Is it flexible enough to allow different activities to trigger different keyboards (or an extra row of custom keys)? Something like Pippy, or Terminal would need that kind of extra flexibility.

Yes - it can be possible to load an extended layout (with for example,
an extra panel on the top for extra characters). It may be a bit
tricky, but sugar can probably provide an API to do this - and it
would be easier if we can wrap libfvkbd in python or extend the
library to use introspection.

>
> - z layering issues with frame, should it be over, under, part of? Currently it can be a mix depending on the sequence things are triggered.

I suppose the frame should always come on top. I'm not sure how the
window manager would deal with this - the window type of the keyboard
panel is currently set to "dock", which can be changed to a window,
and that may work.

>
> - Ideally something (Gnome I assume?) should trigger the keyboard overlay when you focus on a text field, perhaps with some hints about what the 'return' key behaviour should do (or expose a tab key as that is usually the other common text field navigation method). Dismissing the keyboard overlay when a text field is defocused would also be ideal.
>

AFAIK, this requires a GTK+ module to be loaded. I'm still trying to
write a proof of concept implementation of this - it seems that
there's no documentation anywhere for writing GTK+ modules :-(

> - The 'grapes' icon particularly (and some others) could do with with some sugar-love ;) Do you think those should be upstreamed? Or do we have many other unique requirements (enough to fork) that the Meego platform isn't looking to support?

Yeah - that can be got rid of - and it's a part of the layout
specification, so upstreaming it should be a problem.

>
> OT: one thing I really miss on the iPad even after a few weeks solid use, is the omission of cursor keys for small adjustments in text cursor positions or text selections. Text editing, even on an iPad with its auto correction, and realtime frame redraw perfect tap and hold magnifying glass effect can be frustrating. I think cursors are still important keys to have if we expect children to write more than minimal text in this environment.
>
> Sayamindu, what kind'a feedback/assistance would be most useful? Is it too soon to start collating notes and screen shots on a wiki page somewhere?

Yes - I think we should start putting all of this in a wiki.

Thanks,
Sayamindu


-- 
Sayamindu Dasgupta
[http://sayamindu.randomink.org/ramblings]