eBook Reader user interface

[Don Hopkins]

print "Hello world!"

I have made a version of the Read activity that supports navigation and 
control with the gamepad, and has a few new features and improvements.
The main problem is getting the two gamepads to send unique keycodes 
that are not the numeric keypad arrow keys it currently sends (because 
those are already bound to functions that don't do the appropriate thing).
The problem right now is that somebody needs to run the command "chmod 
a+rwx /dev/input/event1" to give Python the permission to perform the 
ioctl that remaps the keys.
The solution to that problem is to decide on a standard set of keycodes 
and have the operating system set them when it boots.
I have tried several sets of keycodes, unsuccessfully. The higher 
keycodes with numbers >512 never seem to make it to the X server.
I tried using the keycodes for the higher numbered function keys 
(KEY_F13 [183] - KEY_F20 [190]), which made it to the X server, but all 
report X keycodes 0.
There may be some magic that needs to be done in the /etc/X11/xorg.conf 
file to admit those keycodes, or maybe some other arcane X server 
configuration.
So for now I am mapping to the letters "esdx" and "ijkm" to test out the 
gamepad code until the OLPC has a fixed set of distinguishable keypad 
keycodes.
It will be very nice to have a gamepad text input method like 
triggerfinger or dasher, to input real text in book mode!

The next issue is rotation. I have made the activity respond to top 
level window resize events by running "xrandr" to figure out which 
direction the screen is rotated.
Then it remaps the behaviors it assigns to the gamepads according to the 
screen rotation.
I think screen rotation should be handled by the application, not the 
operating system, because in some cases the application might want to 
associate commands with the LABELS of the gamepad keys instead of the 
DIRECTIONS of the keys, so it would not want to rotate their meaning.
For example, a tic-tac-toe game that wanted to use X for X and O for O 
would not want to remap the keys on screen rotation! (Admittedly 
contrived, but hey it's a popular game!) c(-;
It would be great if there were a way to find out the screen rotation 
via dbus. Maybe there is but I just haven't figured it out yet. Could we 
add that, if it's not yet supported, please?

Also, I wrote some stuff in the comments of the bug about the screen 
rotation button: it's waaaaay too easy to rotate the screen accidentally 
(not to mention turn the machine off accidentally, ESPECIALLY when the 
screen is upside down and you reach for what you think is the rotation 
button!!!)...
I propose that both the screen rotation button and the power button 
simply pop up big fat obvious dialogs that pop down and do nothing if 
you press the button again, but require you to either press and hold the 
button, or press another button (like the arrow keys) to actually rotate 
the screen or power the system down.
That will let users rotate the screen absolutely in one shot with the 
arrow keys without going through all the intermediate rotations.
Rotating is extremely expensive because it laboriously repaints the 
entire screen, requiring the application to do all kinds of memory 
allocation of giant page sized double buffers. So it should be very hard 
to do it accidentally, while right now it's very hard not to do it 
accidentally.
Of course the consequences of accidentally shutting the system down with 
a mere press of the button are much worse, so the same thing goes 
tenfold for the power button!
Kids will naturally go straight for the buttons on the screen and press 
them many times in a row, so that should not cause any harm, delay or 
distress.

See: Rotating in reverse and/or better feedback for the rotate button
http://dev.laptop.org/ticket/1049

Also: Rotate handheld keys with screen
http://dev.laptop.org/ticket/1443

I decided to use the directional arrows for scrolling withing the page 
selected, for when the entire page doesn't fit on the screen, or when 
you're in continuous mode.
That is because it's possible to press the directional arrows 
diagonally, sending two keycodes at once, which makes sense for 
directional scrolling commands but not for discrete commands.
I used the other 4 button gamepad for discrete user interface navigation 
commands. The left one and the right one move to the previous and next 
selected button, in the same direction as they appear on the screen.
The bottom one activates the selected button. The top one goes to the 
first button. These button mappings rotate with the screen appropriately.

I added a couple buttons for "first page" and "last page", and I also 
added another toolbar of "View" commands, including sizing: fit width, 
best fit, zoom in, zoom out; rotating: rotate left, rotate right; and 
toggle buttons for display mode: dual page, continuous.
When you use the gamepad to navigate between user interface buttons, it 
wraps back and forth from end of one toolbar to the beginning of the other.
Gamepad navigation ignores the "activity" toolbar (for now).
When you navigate with the gamepad, it warps the mouse cursor so it's in 
the upper right corner of the button.
I was hoping that warping the mouse into the button would be enough to 
trigger the tooltip, but apparently not.
I have been looking into how to pop up and tear down the tooltip under 
program control without resorting to ugly hacks like faking events, but 
I can't figure out a way to do it short of hacking the C code.
Any suggestions? The other approach is not to use GTK tooltips at all, 
and implement some nice Sugar tooltips that do the right thing, and 
allow us to add more features like internationalized text (if that's not 
already supported -- I dunno) and icons and animation and nice stuff 
like that.
The tooltips in gamepad mode should be smart enough to pop up when you 
navigate to a button, then go away after a few seconds or after you use 
the button.
If you try to use a button that's "not allowed" (my alternative to gtk's 
broken "not sensitive" which unfortunately disallows tooltips and 
keyboard navigation), a tooltip should pop up explaining why the 
operation is not allowed and what you can do to allow it.
I have written all the code to set the tooltip to the most important 
explanation when buttons are not allowed (i.e. the "back page" button 
says "you need to load a document first" if no doc is loaded, or "you're 
already at the first page" if you do havea doc but you're on the first 
page [duh]).
It's just that the tooltips don't display when I want them to, and 
currently there is no visual indication that the button is not allowed. 
I suppose the "not allowed" state should look just like the "not 
sensitive" state, except that they let you navigate to the button with 
the gamepad and display tooltips.

I made a mixin class for widgets that includes intrinsics for gamepad 
navigation, as well as some other useful stuff (like tooltips, and 
disabling them without breaking keyboard navigation and tooltips).
It also lets (no, requires!) you to associate an "info" dictionary with 
each widget, that tells the system useful stuff about the widget, 
including the icon, default tooltip, and whether or not it's accessible 
via the game controller keypad.
The toolbars have a method that returns a list of their navigable 
widgets, and the application concatenates the widget lists of all 
toolbars, handles keyboard events, and manages to navigate between them.
(This gamepad navigation is in place of the standard GTK tab/arrow key 
navigation, which I consider flawed and too hard-wired to fix, so I just 
programmed around it. For example, on top of the focus and tooltip 
problems, when you arrow to a text field, you're stuck and can't arrow 
out!)
I made the search text field as well as the find next / previous buttons 
inaccessibly from the game controllers, because currently there is no 
way to input text when the keyboard is hidden.
Once we have a gamepad based text input method we can hook that up to 
the search field and enable it for navigation.

I put in placeholder "programmer art" with text labels for the buttons I 
added, which are ugly enough to remind everyone that we need real icons, 
but I'm not a good enough artist to come up with appropriate icons.
 
One usability problem I noticed is with the toggle buttons: the "mouse 
over" highlighting obscures (overrides, conflates, is the same as) the 
highlighting that tells you if the button is selected or not, so once 
you're pointing at the button you have no idea about its current state, 
and no feedback after you click it to change the state, that the state 
was actually changed (modulo the actual effect of the button toggle on 
the application of course). The "mouse over" highlighting should be 
orthogonal to and composeable with the "checked" highlighting. Moving 
over the toggle should activate the "mouse over" highlighting but not 
change the "checked" highlighting. Pressing down and holding should keep 
the "mouse over" highlighting and toggle the "selected" highlighing to 
feedback that it will enter that state iff you release while still in 
the button. If you move out of the button while holding down the mouse, 
the "mouse over" should de-highlight and the "checked" should go back to 
the previous state, to feedback that pressing down and moving out is a 
no-op that cancels the toggle action, and nothing should change when you 
release the button to toggle (but of course you should be able to slide 
back in to resume the "mouse over" and "check toggled" state, then 
release to toggle it.

I heard a rumor that somebody had decided on some unique keycodes for 
the gamepad. Is that true? What are they? Once I know, I'll put them 
into the reader and put up a copy for people to try out. (Right now it 
requires opening a shell and running chmod as root to get it to work, 
which is kinda clumsy, but a great way to force kids to learn Linux, you 
gotta admit.)

    -Don


Zvi Devir wrote:
> I have been collecting a few (more than a few) usability bugs and UI 
> suggestions, which will hopefully be compiled to a list next month.
>
> In the meanwhile, I would like to share a couple of ideas regarding the 
> mapping of the gamepad keys [which I hope won't state the obvious too much].
>
> Currently, the UP/DOWN/LEFT/RIGHT keys have a fixed functionality 
> regardless of the screen orientation. This isn't a correct logic, as the 
> expected functionality changes according to the current orientation. I 
> guess the easiest way to implement this kind of behavior would be using 
> a smart key mapping mechanism in X.
> Another thought -- the current input method treats those keys as 
> direction keys. However, a better input method may be emulating the 
> functionality of the mouse (touch pad) using the gamepad keys. In tablet 
> mode this can really improve the user experience.
>
> Just my $0.04.
>     Zvi
>
>
> Don Hopkins wrote:
>   
>> Goal for improving the eBook reader user interface:
>>
>>   I've been doing some exploratory programming with GTK and Sugar,
>>   trying to improve the user interface of the eBook reader, and make
>>   it useable in book mode with the gamepad.
>>
>>   + Support the game keypads in eBook mode. 
>>
>>     + Low level game keypad support
>>
>>       Need to remap low level keyboard scan codes to Linux keyboard codes. 
>>
>>       The setolpckeys.c program remaps the keys and gamepad buttons.
>>
>> 	Currently it maps both gamepads to the numeric keypad keys (KEY_KP8, etc),
>> 	which the X server and GDK translates to directional keys (GDK_Up, etc).
>>
>> 	I tried to map them to buttons (BTN_A, etc), but the X server seems 
>> 	to ignore keycodes in that range. 
>>
>> 	The xorg.conf file has a keycode mask that looked like it might help, 
>> 	but I couldn't get it to work. 
>>
>> 	Need to have unique keycodes reported for each of the two gamepads, 
>> 	which are not the same as any keyboard keys, without any predefined meanings
>> 	like arrow keys have. 
>>
>> 	Need to define special purpose keycodes just for the OLPC gamepad,
>> 	instead of trying to reuse existing but not appropriate keycodes. 
>>
>> 	What is the process for defining new keycodes in <linux/input.h>?
>>
>> 	Here's my strawman proposal for some new keycodes. 
>>
>> 	  Use keys ("KEY_*") instead of buttons ("BTN_*"), since they
>> 	  seem to work better.
>>
>> 	  The 0x1b* range seems to be unused in <linux/input.h>, 
>> 	  and it's between other groups of keycodes, so I'll
>> 	  propose using that range for the OLPC. 
>>
>> 	  The UP/DOWN/LEFT/RIGHT keys correspond to the directional
>> 	  keypad.
>>
>> 	  #define KEY_XO_GAMEPAD_UP      0x1b0
>> 	  #define KEY_XO_GAMEPAD_DOWN    0x1b1
>> 	  #define KEY_XO_GAMEPAD_LEFT    0x1b2
>> 	  #define KEY_XO_GAMEPAD_RIGHT   0x1b3
>>
>> 	  The NORTH/SOUTH/EAST/WEST keys correspond to the other
>> 	  buttons. Those names are agnostic to the button labels,
>> 	  which may change from the current Playstation buttons
>> 	  (X/O/Triangle/Square). Can anyone suggest better names for
>> 	  the four buttons on the right?
>>
>> 	  #define KEY_XO_GAMEPAD_NORTH   0x1b4
>> 	  #define KEY_XO_GAMEPAD_SOUTH   0x1b5
>> 	  #define KEY_XO_GAMEPAD_EAST    0x1b6
>> 	  #define KEY_XO_GAMEPAD_WEST    0x1b7
>>
>> 	  While we're at it, we could define keycodes for the other
>> 	  OLPC buttons and switches on the screen. I think there are
>> 	  some other sensor switches that could generate keycodes,
>> 	  like opening the screen, rotating it around, and putting it
>> 	  into book mode, so I will make some guesses at names for
>> 	  them, just to get the discussion rolling. 
>>
>> 	  #define KEY_XO_SCREEN_ROTATE   0x1b8
>> 	  #define KEY_XO_SCREEN_POWER    0x1b9
>> 	  #define KEY_XO_SCREEN_OPEN     0x1ba
>> 	  #define KEY_XO_SCREEN_CLOSE    0x1bb
>> 	  #define KEY_XO_SCREEN_IN       0x1bc
>> 	  #define KEY_XO_SCREEN_OUT      0x1bd
>>
>> 	  Is there an exhaustive list of all buttons and switches and
>> 	  events on the OLPC? Are any more planned? Which ones should
>> 	  be assigned keycodes?
>>
>>       Rewrote setolpckeys.c code in Python (just uses ioctl, but needs to know keycodes).
>> 	Writing utilities like that in Python instead of C makes it easier to 
>> 	reconfigure the keys on the OLPC without a C compiler. 
>>
>>     
>
>
> _______________________________________________
> Devel mailing list
> Devel at lists.laptop.org
> http://lists.laptop.org/listinfo/devel
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.laptop.org/pipermail/devel/attachments/20070605/328614c4/attachment.html>