[Sugar-devel] Sugar with a virtual (onscreen) keyboard

[Gary Martin]

Hi Sayamindu,

On 17 Jun 2010, at 13:16, Sayamindu Dasgupta wrote:

> [Apologies for the cross-posting]
> 
> Hello,
> Thanks to the pointers provided by Peter Robinson, I got the Meego
> FVKBD (Free Virtual Keyboard)¹ running along with Sugar.
> A problem with the current FVKBD is that it supports only one base
> layout. Even "variants" of that layout (eg: CapsLock enabled, Symbols,
> etc) are treated as "temporary", which means that you press the "Caps"
> key, enter a capital letter, and immediately after that, it gets reset
> back to the base layout (lower case qwerty).
> I wanted something which would be similar to the existing physical
> keyboards that we ship with the XO machines - with a dedicated key to
> switch between different scripts in the same keyboard. I had to extend
> the code of FVKBD to implement that, and with the modified FVKBD, I
> have spun a live-cd ISO (based on the current SOAS). You can download
> it from http://dev.laptop.org/~sayamindu/sugar-vkbd-test/sugar-vkbd-test.iso

Wow, big thanks for launching into this. For anyone not sure how to try the iso, I'm on a Mac and just used Virtual Box to create a new empty Fedora VM, no HD, and just point to the iso as the boot CD. Started up just fine, keyboard is already open to type in your user name (of course this is all read only, any changes you make will be gone after a reboot).

I'll try and spend some time in the next few days using it via iPad HW and send some feedback, just been playing via mouse so far today.

> Apart from the modified FVKBD, I have added a default keyboard
> definition file which is for English + Bengali, and I've also included
> a sugar device-icon on the frame to control the appearance of the
> keyboard.
> 
> I realize that more needs to be done to support non Latin scripts, and
> here are some of the issues I faced while converting the existing XKB
> Bengali layout:
> 
> * Many scripts do not have a concept of upper case/lower case - so we
> need some other script specific way to divide the characters
> * In the current XKB configurations, non-symbol characters from other
> scripts are often placed in the position of what normally is symbols
> for QWERTY keyboards
> * Numerals pose an interesting problem, since in some places, native
> numerals/digits are quickly being obsoleted, and latin numerals
> (1,2,3..) are becoming the de-facto standard. In these cases, it may
> make sense to provide only _one_ layout/state for numerals, and allow
> users to input native numerals by hovering (touch + hold) on the
> virtual key for the latin digit.
> 
> Among the general issues, I'm not sure how to deal with the keyboard
> taking up half of the screen real estate - it may be worthwhile to see
> if we can have a "split screen" sort of configuration while the
> keyboard is active.

It didn't bother me too much, and this was in an 800x600 session, though ideally we would want the text insertion point to be visible above the keyboard (FWIW various iPad apps have different success in dealing with this, all of Apple's are fine, but it seems 3rd parties do need to do some work on the app side to keep this behaviour working at all times).

> Thoughts, feedback, etc would be appreciated :-).

Yes, lot's of interesting items to cover :-) I'll try to start to put together a list. Some quick item that struck me right away:

- the Meego keyboard design is clearly for casual typing/text entry, no way of typing commands or many symbols needed for basic programming work – diving into terminal to use vi, or worse emacs, is pretty much a dead end (unless ctrl and alt keys are hidden somewhere I couldn't find). Is it flexible enough to allow different activities to trigger different keyboards (or an extra row of custom keys)? Something like Pippy, or Terminal would need that kind of extra flexibility.

- z layering issues with frame, should it be over, under, part of? Currently it can be a mix depending on the sequence things are triggered.

- Ideally something (Gnome I assume?) should trigger the keyboard overlay when you focus on a text field, perhaps with some hints about what the 'return' key behaviour should do (or expose a tab key as that is usually the other common text field navigation method). Dismissing the keyboard overlay when a text field is defocused would also be ideal.

- The 'grapes' icon particularly (and some others) could do with with some sugar-love ;) Do you think those should be upstreamed? Or do we have many other unique requirements (enough to fork) that the Meego platform isn't looking to support?

OT: one thing I really miss on the iPad even after a few weeks solid use, is the omission of cursor keys for small adjustments in text cursor positions or text selections. Text editing, even on an iPad with its auto correction, and realtime frame redraw perfect tap and hold magnifying glass effect can be frustrating. I think cursors are still important keys to have if we expect children to write more than minimal text in this environment.

Sayamindu, what kind'a feedback/assistance would be most useful? Is it too soon to start collating notes and screen shots on a wiki page somewhere?

Regards,
--Gary