My JavaScript book is out! Don't miss the opportunity to upgrade your beginner or average dev skills.

Tuesday, June 19, 2012

Dealing With Future Pointers

I have talked about this already in both JSConfEU and QCon - London and I have also blogged a couple of times about this problem that many developers keep ignoring ...

The Unexpected Input


In JSConfEU I have showed with my slides this picture:
iPad and magic mouse
which is simply the reason most of the so called mobile websites won't work as expected.
Why that? Because "ontouchend" in window, the most pointless feature detection ever, will produce a positive result and if you decide which event should be attached to the document or any DOM node relying this check, the moment the device that exposes touch events BUT has a trackpad, mouse, or any other alternative pointer device connected, it will fail, it won't work, it will look broken!

A Hybrid Solution

We could make the assumption that if the user is using fingers, the user will keep using fingers ... while if the user chooses the alternative pointer, there shouldn't be a case where fingers are again on the screen.
For this kind of situation/assumption, a lazy pointer detection is the only one we can rely.

// useless/unoptimized code - used to explain and nothing else
function initPointerEvents(e) {
// decide what should be used later on
switch (e.type) {
case "touchstart": return useTouches();
// other possible cases
default: return useMouse();
}
document.removeEventListener("touchstart", initPointerEvents, true);
document.removeEventListener("mousedown", initPointerEvents, true);
document.removeEventListener("mousemove", initPointerEvents, true);
document.removeEventListener("mousewheel", initPointerEvents, true);
}
// note the usage of true, so that we can be sure these detections are performed
// before any other listener attached with "false" ( bubbling phase )
document.addEventListener("touchstart", initPointerEvents, true);
document.addEventListener("mousedown", initPointerEvents, true);
document.addEventListener("mousemove", initPointerEvents, true);
document.addEventListener("mousewheel", initPointerEvents, true);

With above example it becomes really easy to understand what's the user choice: either touch, or keyboard.
No matter what kind of device we are dealing with, the very first event will tell us what kind/group of events we should use.

Well ... Still Not Friendly

What if the user combines touches and trackpad?
Microsoft Surface Tablet
The Microsoft Surface proposal looks great to me, but at the same time it might fail in most common mobile oriented websites.
First of all, MS has introduced kinda proprietary events to deal with pointers but I wonder how these events will behave once the user decide to switch between trackpad and screen.
I imagine that the mouse cursor will appear and disappear accordingly, at least that is what could make sense to me, so that the page could actually implement both :hover styles and still be based on touch events but the simple fact is that all this is a mess for web pages.
Here an example of how things could go wrong, assuming the classic trackpad in older laptop won't mean that msPointerEnabled is true ( being that screen not touchable )

// the MS suggested feature detection
if (window.navigator.msPointerEnabled) {
// fails when the user uses the trackpad instead of the screen
} else {
// fails when the user uses the screen instead of the trackpad
}

So, unless Windows 8 does not prefer MS events regardless the hardware, the inline feature detection will fail in this case as well.

We Need A Better Interface

This call is mainly for W3C, everything we have right now does not scale with more complex, modern, hybrid, devices.
This is also not only about mouses, trackpads, and screens, this is related to virtual keyboards too, the most annoying thing could appear and without notification/control in a webpage screen.
What we could do is to create yet another library able to deal in a totally abstract way with all these cases and in real time.
But how big this overhead would be? Think how many times we attach events to single nodes, rather than into the document only, and think how many checks we need to do in order to normalize properly and cross platform user actions.
The ideal library should be able to switch runtime and handle combination of any sort of pointer ... maybe you scroll a page with fingers, then you draw on canvas with a USB pen ... then you type on the screen, then suddenly you press enter in the keyboard ... and so on ...
Asus Transformer Prime


One Problem At The Time

There is no library out there able to truly behave as expected and switch runtime all these possibilities ... so I might decide to write one but the architecture should be both simple and able to scale.
I am not a huge fun of too abstract architectures for the simple reason that these rarely have good performances but this is a problem that later we deal with, more problems we'll have.
If you have any library I am not aware of that is that smart, please let me know and I'll update the post linking to this library.

4 comments:

Dom said...

This sounds like what the W3C IndieUI Working Group is trying to address: http://www.w3.org/WAI/IndieUI/ "an abstraction between device-specific user interaction events and inferred user intent such as scroll, activate, etc. This provides an intermediate layer between device- and modality-specific user interaction events, and the basic user interface functionality used by Web applications"

Sander said...

I wholheartly agree. This needs to be tackled ASAP.
Isn't this the same problem Boris Smus was addressering in his pointer.js.
Perhaps you can contact him.
this is the library he proposed: https://github.com/borismus/pointer.js

Andrea Trasatti said...

Andrea,
this post is old, so probably a couple of things are not up-to-date. Let me record them here for future readers.

Microsoft submitted the Pointer Events spec to the W3C, it is currently under review by Google and Mozilla, we can expect comments, but the good news is that it looks like everyone agrees it's a good spec!

The Pointer events spec already takes into account that a user might switch from one input form to another. In my understanding it assumes that you keep using that same input form for the entire action (i.e. I am not sure what happens if I use the mouse with one hand to do a swipe and at the same time touch the screen with my other hand). The msPointerEnabled (and its unprefixed counterpart) simply tells you if the browser implements the standard, not WHICH pointer is being used.

The spec also defines which is the "primary pointer" for the current device, i.e. defines a sort of hierarchy for mouse, touch, pen. It is also designed to be extensible for future input forms.

IMHO, everything that you describe in this article is covered by the pointer events, but I am happy to discuss further and possibly help you craft a comment to send to the W3C for improvements.

Andrea Giammarchi said...

thanks Andrea, the thing I didn't know/mention here is that there is a polyfill that works quite well ... PointerEvents