New Apple patent shows AR/VR headset with gesture and expression tracking

“The Apple mixed reality headset will feature several sensors on the outside to track expressions, gestures, and eyes of the user” 

Apple recently filed for a new patent for a mixed reality headset with sensors. The company filed the patent in March this year and it is titled ‘Display System Having Sensors’. The patent was published by the USPTO only last week and it describes the display system having several sensors that will be able to track eyes, facial expressions, and gestures. These inputs would be combined with the sensors on the outside to provide a mixed reality experience to the user. The display inside the headset will be used to display the outside view in high resolution, which should add to the experience of mixed reality.

Notably, Apple has been tipped to be working on a mixed reality headset. A report from March, published by analyst Ming-Chi Kuo, stated that the company would release such a product in 2020. It also detailed that the headset would connect to an iPhone wirelessly. The iPhone would be required to make the headset work, as it would use the processing power, GPS, and network of the phone. He also reported that the headset would run on a custom OS and production will start in Q2 2020. 

The Apple patent shows cameras placed outside the headset to relay images on to a display inside.
The Apple patent shows cameras placed outside the headset to relay images on to a display inside.

Now, the recently published patent doesn’t really tell us if the headset will connect to an iPhone, but it does describe the workings of the sensors. As discussed above, the sensors could be used to produce a realistic representation of users’ faces in mixed reality. Some of the sensors will record jaw and eyebrow movements, while others will track the eye. The headset could also use eye movements for biometric authentication.

The headset will be able to overlay objects in the real world, similar to how AR works on the current iPhones and iPads. However, the experience would feel more real with the headset. Cameras, or world sensors as Apple calls it, will be placed outside the headset to relay images to a display inside in high quality. This is very different from how the mixed reality headsets from Microsoft or Leap work. We will have to wait a while to find out more about this Apple headset. It is likely coming, but we don’t have any definite dates yet. Hopefully, Apple gives us some information at the iPhone launch later this year.