Scientists from Reality Labs at Meta have invented a device that allows users to interact with computers using only hand gestures.

The device sits like a bracelet on the wrist where it senses electrical signals generated by muscle movements. It then translates these into commands which are transmitted to the computer via Bluetooth.

Keyboards, touchpads and computer mice have been the default means of interacting with computers for decades, but not everyone has the physical ability to use them. People with smartphones, smartwatches and smartglasses may also need new ways to control their devices without having to shift their attention to a touch screen.

The new “neuromotor interface” could address these problems.

“We chose the wrist because humans primarily engage the world with their hands, and the wrist provides broad coverage of sEMG signals of hand, wrist and forearm muscles,” the researchers write in a paper presenting the technology in the journal Nature.

The device relies on “surface electromyography” (sEMG) which detects muscle activity via metal electrodes on the skin.

The bracelet is non-invasive and easy to put on and remove unlike previous forays into brain-computer interfaces which interact directly with brain tissue or require lengthy setups.

The researchers recruited more than 6,600 participants to perform 3 different virtual tasks while wearing the sEMG device.

In the first task they controlled a computer cursor using the angle of their wrists. In the second, they were prompted to perform 9 different hand gestures in a randomised order. The final handwriting task asked participants to pinch their fingers together, as if holding an imaginary pen, to pretend to write out sentence prompts.

This data was then used to train deep learning models to translate sEMG signals into their intended computer inputs.

Importantly, the large training dataset allowed the models to accurately interpret commands from lots of people. They can account for differences in anatomy, physiology, and behaviour without the need for time-consuming calibration steps.

After training, a different group of participants were able to complete the virtual tasks using only the sEMG decoders, though they performed better while using conventional interfaces.

The authors say these “cannot fulfil the same role as an always available sEMG wristband”.

“They require cumbersome equipment: tracking wrist angles requires multiple calibrated cameras, using a laptop trackpad or a gaming controller encumbers the hand, and handwriting requires a pen, paper and a surface,” they write.

“For tasks in which constant availability is important (such as on-the-go scenarios), the reductions in current decoder performance may therefore be acceptable.”

They add that sEMG decoding will likely improve further through innovations in sensing performance, better models, and the development of greater user skill over time. “We expect user proficiency to grow with increased familiarity with the sEMG-RD and underlying gestures.”

The wristband may also offer a way for people with limited movement to communicate with computers.

“In the clinic, the ability to design interactions that require only minimal muscular activity, rather than performance of a specific movement, could enable viable interaction schemes for those with reduced mobility, muscle weakness or missing effectors entirely,” they write.

“It is unclear whether the generalised models developed here and trained on able-bodied participants will be able to generalise to clinical populations, although early work appears promising.

“New applications will be facilitated by … increasingly diverse datasets covering populations with motor disabilities and potentially combining with other signals recorded at the wrist.”