I think his point was that it would be a bit silly to put this kind of solution into a product -- which is, in a word: jank -- when capacitive touchscreens are already a tried and true solution in production devices.
I often think “hand-pointing” interfaces have unexplored / unrealised potential... like, who would want a touch TV screen?! Sure, you could have the touch UX for the TV on your phone... but even better would be if you would be able to just point to the TV and MAGIC. The biggest problem I see is that humans aren’t that good at pointing - we can only “aim” with one eye, keeping both eyes open messes it up.
If you had feedback for where you were pointing I think most people would get pretty good pretty quickly (excepting people with movement disorders). It wouldn't be very precise, but I think it would be great for lazily triggering multitouch things like zoom/scroll/page-up/page-down/switch-app etc.
I've often wondered what having one of these on a laptop, tracking space above the keyboard / in front of the screen, would be like: https://www.leapmotion.com/
Gestural stuff is nice when it's transparent and guess-able. Sadly not often the case, but when it is it's pretty magical.
I tried doing this with the internal webcam, using a clip-on fisheye lens, and mirrors. And eventually punted, using the bare webcam only for head tracking, and adding usb cameras on sticks perched on the laptop screen. With more sticks to protect the usb sockets from the cables. And lots of gaff tape.
> leapmotion
Leap Motion has finally been acquired, so the future of the product is unclear. And it's Windows-only (the older and even cruftier version supporting linux, doesn't do background rejection, and so can't be used pointing down at a keyboard). But it has apis, so you can export the data. My fuzzy impression is it's not quite good enough for surface touch events, but it's ok-ish for pose and gestures. When the poses don't have a lot of occlusion. And the device is perched on a stick in front of your face.
> Gestural stuff is nice when it's transparent and guess-able. Sadly not often the case
I fuzzily recall some old system (Lisp Machine?) as having a status bar with a little picture of a mouse, and telling what its buttons would do if pressed. And a key impact of VR/AR is having more and cheaper UI real estate to work with. So always showing what gestures are currently available, and what they do, should become feasible.
Even on a generic laptop screen, DIYed for 3D, it seems you might put such secondary information semitransparently above the screen plane. And sort of focus through it to work. Making the overlay merely annoying, rather than intolerable.
But when it all works, yeah, magical. Briefly magical. The future is already here... it just has a painfully low duty cycle. And ghastly overhead.
I've often wanted some eye tracking device when working on more than one screen. Like, when looking at my terminal screen I just want to start typing, instead of alt-tab'ing to it first.
I actually saw a demo of this (plus more; it was a “universal remote”) at Cal Hacks a few years ago. They used a Myo for gesture recognition and the whole thing was really slick.
WRONG! I had a working touch screen in my Dell Laptop, but when I went on vacation to Cuba the screen mouse moved a random. I noticed it worked properly in my room after half an hour or early in the morning if I was outside.
But once it it got warm and humid it was unusable. I had to turn off the touchscreen and then I could at-less use the mouse.
