Augmented reality (AR) and mixed reality (MR) headsets let us see the world around us with virtual elements superimposed on top. For example, many modern AR/MR headsets use hand-tracking cameras to detect hand gestures in the air, which allows users to type on a virtual keyboard that appears to be floating in front of the user. As exciting as this sounds initially, the approach often needs handheld controllers, and keeping your hands in the air for a long time can cause arm fatigue. These air gestures also lack physical feedback, which can make interaction less comfortable.

Searching for an innovative way to make typing more comfortable while using AR/MR technology, we found a method that transforms regular surfaces into keyboards - no extra equipment required.

The key to this innovation is a natural reaction our body has when we apply pressure to our fingertips. If you try pressing down on a hard surface like a desk, you will notice a flush of white. This color change is called the blanching phenomenon. By analyzing images from the headset camera with an AI model, the system can detect when a fingertip touches a surface.

We trained an AI model to reliably recognize the blanching phenomenon in fingertip images captured by a standard camera on the headset. In addition, a supporting interaction system was developed to provide easy-to-access user experience.

User studies showed that the system reliably detected fingertip contact across several common surface materials. Participants were able to perform interaction tasks with stable input accuracy. The surface-supported interaction also allowed users to rest their fingers on the surface during operation.