Quest Pro turns your eyes into a keyboard

Picture: Fabio Dela Antonio

Der Artikel kann nur mit aktiviertem JavaScript dargestellt werden. Bitte aktiviere JavaScript in deinem Browser and lade die Seite neu.

Meta Quest Pro opens up new possibilities. A developer shows how you can write in virtual reality using eye tracking.

In a video posted to YouTube and other channels, Fabio Dela Antonio, iOS developer and XR enthusiast, demonstrates an eye-tracking experiment. Antonio’s eyes scan each key of a virtual keyboard. Little by little, the phrase “Hello World” emerges. Antonio selects characters by eye tracking on a Quest Pro (review).

The seizure occurs as soon as the gaze lands on a virtual key for half a second. The developer does not need more than his eyes. In the video, Antonio holds his hands out in front of him to prove he’s not secretly typing on a keyboard after all.

“Blipping” is not a keyboard replacement

As interesting as the idea of ​​gaze-controlled striking or blipping may seem, it is unlikely to replace physical striking. Typing just takes too long for that. For example, the 12 seconds it took Antonio to type the words “Hello World”.

It would be a little faster if a controller key could be used for confirmation. However, preserves the second problem of this approach: the need to look at a keyboard while typing. This is no longer necessary with physical input, which also allows fast input in the case of the ten-finger system.

Such a feature would still be useful, however. For example, as an alternative to heavier input via the VR controller and virtual laser pointer, which is still the standard input method in Meta Quest 2 (review) and Pro.

A big win would be gaze-controlled typing for people who can’t use their hands to type due to a disability. This is exactly the group Antonio had in mind. He hopes that Meta will bring such accessibility feature to MetaQuest Pro.


Looking for the XR keyboard

Effective typing in VR and AR is a problem not solved. In addition to the laser pointer approach mentioned above, Meta Quest 2 & Pro has the ability to bring the physical keyboard to VR. This only partially solves the problem. After all, we want to do without a real keyboard one day. For example, using a wristband that turns any surface into a keyboard.

Meta is pursuing an AI-supported finger-tracking keyboard for this purpose and has also demonstrated a system called “Pinchtype”, in which inputs come from pinch gestures. This approach is innovative but requires practice.

Speaking of artificial intelligence, the Slide-To-Type System known from smartphones, where you slide your fingers across the keyboard and an AI model guesses the desired word, could simplify and speed up Antonio’s gaze-controlled typing. In this case, it may be enough to let your gaze wander over the sequence of letters.

You can download its prototype from Github and install it on Meta Quest Pro via Sidequest (Guide).

Leave a Comment

Your email address will not be published. Required fields are marked *