[sugar] eBook Reader user interface

[Dan Williams]

On Tue, 2007-06-05 at 12:38 -0700, Don Hopkins wrote:
> print "Hello world!"
> 
> I have made a version of the Read activity that supports navigation
> and control with the gamepad, and has a few new features and
> improvements. 
> The main problem is getting the two gamepads to send unique keycodes
> that are not the numeric keypad arrow keys it currently sends (because
> those are already bound to functions that don't do the appropriate
> thing). 
> The problem right now is that somebody needs to run the command "chmod
> a+rwx /dev/input/event1" to give Python the permission to perform the
> ioctl that remaps the keys. 
> The solution to that problem is to decide on a standard set of
> keycodes and have the operating system set them when it boots. 
> I have tried several sets of keycodes, unsuccessfully. The higher
> keycodes with numbers >512 never seem to make it to the X server. 
> I tried using the keycodes for the higher numbered function keys
> (KEY_F13 [183] - KEY_F20 [190]), which made it to the X server, but
> all report X keycodes 0. 
> There may be some magic that needs to be done in
> the /etc/X11/xorg.conf file to admit those keycodes, or maybe some
> other arcane X server configuration. 
> So for now I am mapping to the letters "esdx" and "ijkm" to test out
> the gamepad code until the OLPC has a fixed set of distinguishable
> keypad keycodes. 
> It will be very nice to have a gamepad text input method like
> triggerfinger or dasher, to input real text in book mode! 
> 
> The next issue is rotation. I have made the activity respond to top
> level window resize events by running "xrandr" to figure out which
> direction the screen is rotated. 

I think the plan here is to have Sugar or the hardware manager send out
events so that activities know when the screen is rotated.

> Then it remaps the behaviors it assigns to the gamepads according to
> the screen rotation. 
> I think screen rotation should be handled by the application, not the
> operating system, because in some cases the application might want to
> associate commands with the LABELS of the gamepad keys instead of the
> DIRECTIONS of the keys, so it would not want to rotate their meaning. 

So basically, you mean not _rotation_ handled by the activity, but
allowing the game button meanings to be specified by the activity
depending on variables like rotation, right?

> For example, a tic-tac-toe game that wanted to use X for X and O for O
> would not want to remap the keys on screen rotation! (Admittedly
> contrived, but hey it's a popular game!) c(-;
> It would be great if there were a way to find out the screen rotation
> via dbus. Maybe there is but I just haven't figured it out yet. Could
> we add that, if it's not yet supported, please? 

Yes, that's the way it should work.

> Also, I wrote some stuff in the comments of the bug about the screen
> rotation button: it's waaaaay too easy to rotate the screen
> accidentally (not to mention turn the machine off accidentally,
> ESPECIALLY when the screen is upside down and you reach for what you
> think is the rotation button!!!)... 

The power button functionality will be changed to put the machine to
sleep rather than power off.  We may have a 2-second timeout before it
sleeps though, to alleviate this problem further.  Note that it's the EC
that dictates the 4-second poweroff when holding down the power button,
and we can't change that behavior easily.  We can make momentary presses
of the power button, or holding it down up to 4 seconds, do anything we
want fairly easily though.

> I propose that both the screen rotation button and the power button
> simply pop up big fat obvious dialogs that pop down and do nothing if 

Dialogs aren't the right solution here.

> you press the button again, but require you to either press and hold
> the button, or press another button (like the arrow keys) to actually
> rotate the screen or power the system down.

That's pretty indirect; I'd argue the more direct the better.  The
button has a rotation icon, and when you press it, it should rotate the
screen.

> That will let users rotate the screen absolutely in one shot with the
> arrow keys without going through all the intermediate rotations. 
> Rotating is extremely expensive because it laboriously repaints the
> entire screen, requiring the application to do all kinds of memory
> allocation of giant page sized double buffers. So it should be very
> hard to do it accidentally, while right now it's very hard not to do
> it accidentally. 

This is not a problem with the B3 units and the LX processor, because
the LX is (a) faster and (b) can accelerate resizes.  It's so damn quick
on the LX you'll be literally blown away after seeing it on a B1/B2.
Obviously the activity has to redraw; but that should already be as fast
as possible.

> Of course the consequences of accidentally shutting the system down
> with a mere press of the button are much worse, so the same thing goes
> tenfold for the power button! 

Again, slated to be fixed.

> Kids will naturally go straight for the buttons on the screen and
> press them many times in a row, so that should not cause any harm,
> delay or distress. 
> 
> See: Rotating in reverse and/or better feedback for the rotate button
> http://dev.laptop.org/ticket/1049
> 
> Also: Rotate handheld keys with screen
> http://dev.laptop.org/ticket/1443
> 
> I decided to use the directional arrows for scrolling withing the page
> selected, for when the entire page doesn't fit on the screen, or when
> you're in continuous mode. 
> That is because it's possible to press the directional arrows
> diagonally, sending two keycodes at once, which makes sense for
> directional scrolling commands but not for discrete commands. 
> I used the other 4 button gamepad for discrete user interface
> navigation commands. The left one and the right one move to the
> previous and next selected button, in the same direction as they
> appear on the screen.

Eben has some ideas on what the game buttons should mean when the
keyboard isn't available (ie, ebook mode).  We may want to standardize
some of the buttons so that you can do things like switch activities
while in ebook mode, which would mean assigning some of the game
keycodes to standard Sugar actions, not activity actions.

> The bottom one activates the selected button. The top one goes to the
> first button. These button mappings rotate with the screen
> appropriately. 
> 
> I added a couple buttons for "first page" and "last page", and I also
> added another toolbar of "View" commands, including sizing: fit width,
> best fit, zoom in, zoom out; rotating: rotate left, rotate right; and
> toggle buttons for display mode: dual page, continuous. 

We've got to keep in mind that the default UI should be as simple as
possible, which should be the case here even with these additions.  ie,
you don't have to care about any of the options above unless you really
need to.  That's what nice about the toolbars; they aren't there and
clickable when you open the activity.  Just keep in mind that for things
like the reader, the basic UI should keep out of the way until you
really need the function, which should then be easy to get to.

> When you use the gamepad to navigate between user interface buttons,
> it wraps back and forth from end of one toolbar to the beginning of
> the other. 
> Gamepad navigation ignores the "activity" toolbar (for now). 
> When you navigate with the gamepad, it warps the mouse cursor so it's
> in the upper right corner of the button.
> I was hoping that warping the mouse into the button would be enough to
> trigger the tooltip, but apparently not. 
> I have been looking into how to pop up and tear down the tooltip under
> program control without resorting to ugly hacks like faking events,
> but I can't figure out a way to do it short of hacking the C code. 
> Any suggestions? The other approach is not to use GTK tooltips at all,
> and implement some nice Sugar tooltips that do the right thing, and
> allow us to add more features like internationalized text (if that's
> not already supported -- I dunno) and icons and animation and nice
> stuff like that. 
> The tooltips in gamepad mode should be smart enough to pop up when you
> navigate to a button, then go away after a few seconds or after you
> use the button.
> If you try to use a button that's "not allowed" (my alternative to
> gtk's broken "not sensitive" which unfortunately disallows tooltips
> and keyboard navigation), a tooltip should pop up explaining why the
> operation is not allowed and what you can do to allow it. 
> I have written all the code to set the tooltip to the most important
> explanation when buttons are not allowed (i.e. the "back page" button
> says "you need to load a document first" if no doc is loaded, or
> "you're already at the first page" if you do havea doc but you're on
> the first page [duh]). 
> It's just that the tooltips don't display when I want them to, and
> currently there is no visual indication that the button is not
> allowed. I suppose the "not allowed" state should look just like the
> "not sensitive" state, except that they let you navigate to the button
> with the gamepad and display tooltips. 
> 
> I made a mixin class for widgets that includes intrinsics for gamepad
> navigation, as well as some other useful stuff (like tooltips, and
> disabling them without breaking keyboard navigation and tooltips). 
> It also lets (no, requires!) you to associate an "info" dictionary
> with each widget, that tells the system useful stuff about the widget,
> including the icon, default tooltip, and whether or not it's
> accessible via the game controller keypad. 

interesting; sounds useful

> The toolbars have a method that returns a list of their navigable
> widgets, and the application concatenates the widget lists of all
> toolbars, handles keyboard events, and manages to navigate between
> them. 
> (This gamepad navigation is in place of the standard GTK tab/arrow key
> navigation, which I consider flawed and too hard-wired to fix, so I
> just programmed around it. For example, on top of the focus and
> tooltip problems, when you arrow to a text field, you're stuck and
> can't arrow out!) 
> I made the search text field as well as the find next / previous
> buttons inaccessibly from the game controllers, because currently
> there is no way to input text when the keyboard is hidden. 

Also, B3 units have replaced the Camera key with a Search key that when
pressed should automatically focus the search field or something like
that.

> Once we have a gamepad based text input method we can hook that up to
> the search field and enable it for navigation. 
> 
> I put in placeholder "programmer art" with text labels for the buttons
> I added, which are ugly enough to remind everyone that we need real
> icons, but I'm not a good enough artist to come up with appropriate
> icons. 
>  
> One usability problem I noticed is with the toggle buttons: the "mouse
> over" highlighting obscures (overrides, conflates, is the same as) the
> highlighting that tells you if the button is selected or not, so once
> you're pointing at the button you have no idea about its current
> state, and no feedback after you click it to change the state, that
> the state was actually changed (modulo the actual effect of the button
> toggle on the application of course). The "mouse over" highlighting
> should be orthogonal to and composeable with the "checked"
> highlighting. Moving over the toggle should activate the "mouse over"
> highlighting but not change the "checked" highlighting. Pressing down
> and holding should keep the "mouse over" highlighting and toggle the
> "selected" highlighing to feedback that it will enter that state iff
> you release while still in the button. If you move out of the button
> while holding down the mouse, the "mouse over" should de-highlight and
> the "checked" should go back to the previous state, to feedback that
> pressing down and moving out is a no-op that cancels the toggle
> action, and nothing should change when you release the button to
> toggle (but of course you should be able to slide back in to resume
> the "mouse over" and "check toggled" state, then release to toggle
> it. 

Check out how GTK theme states are done; there are 5 states and each
should be visibly independent of the others.  They are something like
DISABLED, NORMAL, SELECTED, PRELIGHT, and one I forget.  It's quite
possible that the Sugar GTK theme doesn't implement all of these
correctly.

> I heard a rumor that somebody had decided on some unique keycodes for
> the gamepad. Is that true? What are they? Once I know, I'll put them
> into the reader and put up a copy for people to try out. (Right now it
> requires opening a shell and running chmod as root to get it to work,
> which is kinda clumsy, but a great way to force kids to learn Linux,
> you gotta admit.) 

Jim Gettys is dealing with the X keycodes; but we're busy getting Xorg
1.3/(X 7.3) into the builds so that we can actually do more flexible
input stuff like all this requires.

Dan

>     -Don
> 
> 
> Zvi Devir wrote: 
> > I have been collecting a few (more than a few) usability bugs and UI 
> > suggestions, which will hopefully be compiled to a list next month.
> > 
> > In the meanwhile, I would like to share a couple of ideas regarding the 
> > mapping of the gamepad keys [which I hope won't state the obvious too much].
> > 
> > Currently, the UP/DOWN/LEFT/RIGHT keys have a fixed functionality 
> > regardless of the screen orientation. This isn't a correct logic, as the 
> > expected functionality changes according to the current orientation. I 
> > guess the easiest way to implement this kind of behavior would be using 
> > a smart key mapping mechanism in X.
> > Another thought -- the current input method treats those keys as 
> > direction keys. However, a better input method may be emulating the 
> > functionality of the mouse (touch pad) using the gamepad keys. In tablet 
> > mode this can really improve the user experience.
> > 
> > Just my $0.04.
> >     Zvi
> > 
> > 
> > Don Hopkins wrote:
> >   
> > > Goal for improving the eBook reader user interface:
> > > 
> > >   I've been doing some exploratory programming with GTK and Sugar,
> > >   trying to improve the user interface of the eBook reader, and make
> > >   it useable in book mode with the gamepad.
> > > 
> > >   + Support the game keypads in eBook mode. 
> > > 
> > >     + Low level game keypad support
> > > 
> > >       Need to remap low level keyboard scan codes to Linux keyboard codes. 
> > > 
> > >       The setolpckeys.c program remaps the keys and gamepad buttons.
> > > 
> > > 	Currently it maps both gamepads to the numeric keypad keys (KEY_KP8, etc),
> > > 	which the X server and GDK translates to directional keys (GDK_Up, etc).
> > > 
> > > 	I tried to map them to buttons (BTN_A, etc), but the X server seems 
> > > 	to ignore keycodes in that range. 
> > > 
> > > 	The xorg.conf file has a keycode mask that looked like it might help, 
> > > 	but I couldn't get it to work. 
> > > 
> > > 	Need to have unique keycodes reported for each of the two gamepads, 
> > > 	which are not the same as any keyboard keys, without any predefined meanings
> > > 	like arrow keys have. 
> > > 
> > > 	Need to define special purpose keycodes just for the OLPC gamepad,
> > > 	instead of trying to reuse existing but not appropriate keycodes. 
> > > 
> > > 	What is the process for defining new keycodes in <linux/input.h>?
> > > 
> > > 	Here's my strawman proposal for some new keycodes. 
> > > 
> > > 	  Use keys ("KEY_*") instead of buttons ("BTN_*"), since they
> > > 	  seem to work better.
> > > 
> > > 	  The 0x1b* range seems to be unused in <linux/input.h>, 
> > > 	  and it's between other groups of keycodes, so I'll
> > > 	  propose using that range for the OLPC. 
> > > 
> > > 	  The UP/DOWN/LEFT/RIGHT keys correspond to the directional
> > > 	  keypad.
> > > 
> > > 	  #define KEY_XO_GAMEPAD_UP      0x1b0
> > > 	  #define KEY_XO_GAMEPAD_DOWN    0x1b1
> > > 	  #define KEY_XO_GAMEPAD_LEFT    0x1b2
> > > 	  #define KEY_XO_GAMEPAD_RIGHT   0x1b3
> > > 
> > > 	  The NORTH/SOUTH/EAST/WEST keys correspond to the other
> > > 	  buttons. Those names are agnostic to the button labels,
> > > 	  which may change from the current Playstation buttons
> > > 	  (X/O/Triangle/Square). Can anyone suggest better names for
> > > 	  the four buttons on the right?
> > > 
> > > 	  #define KEY_XO_GAMEPAD_NORTH   0x1b4
> > > 	  #define KEY_XO_GAMEPAD_SOUTH   0x1b5
> > > 	  #define KEY_XO_GAMEPAD_EAST    0x1b6
> > > 	  #define KEY_XO_GAMEPAD_WEST    0x1b7
> > > 
> > > 	  While we're at it, we could define keycodes for the other
> > > 	  OLPC buttons and switches on the screen. I think there are
> > > 	  some other sensor switches that could generate keycodes,
> > > 	  like opening the screen, rotating it around, and putting it
> > > 	  into book mode, so I will make some guesses at names for
> > > 	  them, just to get the discussion rolling. 
> > > 
> > > 	  #define KEY_XO_SCREEN_ROTATE   0x1b8
> > > 	  #define KEY_XO_SCREEN_POWER    0x1b9
> > > 	  #define KEY_XO_SCREEN_OPEN     0x1ba
> > > 	  #define KEY_XO_SCREEN_CLOSE    0x1bb
> > > 	  #define KEY_XO_SCREEN_IN       0x1bc
> > > 	  #define KEY_XO_SCREEN_OUT      0x1bd
> > > 
> > > 	  Is there an exhaustive list of all buttons and switches and
> > > 	  events on the OLPC? Are any more planned? Which ones should
> > > 	  be assigned keycodes?
> > > 
> > >       Rewrote setolpckeys.c code in Python (just uses ioctl, but needs to know keycodes).
> > > 	Writing utilities like that in Python instead of C makes it easier to 
> > > 	reconfigure the keys on the OLPC without a C compiler. 
> > > 
> > >     
> > 
> > 
> > _______________________________________________
> > Devel mailing list
> > Devel at lists.laptop.org
> > http://lists.laptop.org/listinfo/devel
> >   
> 
> _______________________________________________
> Sugar mailing list
> Sugar at lists.laptop.org
> http://lists.laptop.org/listinfo/sugar