Prophesee, a pioneer and market leader of event-based neuromorphic vision technology recently announced a collaboration with Tobii, the global leader in eye tracking and attention computing. The goal is to deliver a next-generation event-based eye tracking solution tailored for AR/VR and smart eyewear applications. Given the discussion above on the fusion of IVAS, Meta and Anduril capabilities for AR/VR headsets, this is an exciting development. Event based sensing asynchronously reads out only the pixels which record intensity changes above a user defined threshold, resulting in lower latency, power, compute, storage and thermal management requirements.

Prophesee (“predicting and seeing where the action is”), based in France, uses its event-based cameras for industrial automation, AR/VR, security, healthcare and AoT™ (Autonomy of Things) applications. Founded in 2014, the company closed its C round funding of $50M in October 2022. To date, it has raised $127M. Prophesee’s focus is on providing its sensor chips, SDK (Software Development Kit) and camera reference designs to end users like Tobii and others who develop imaging systems for various applications. It’s next generation MetaVision GenX 320 QVGA sensor (320 x 320 pixels, 6 μm pixel pitch, 4 mm x 3 mm footprint) is designed for eye tracking in gaming and wearable AR/VR headsets. Figure 1 highlights some of the capabilities:

Luca Verra is the CEO of Prophesee. A key initiative underway is the development of an even leaner event sensing chip than the MetaVision QVGA for smart glass applications. Reducing the pixel pitch by 50% (< 3 um) realizes a 5X reduction in the chip area. While the MetaVision sensor consumes ~25 mW of power at a system level (includes the sensing readout, signal pre-processing and a 900 um wavelength LED), the dedicated chip for smart glasses aims to reduce power consumption by > 10X, and provide data speeds approaching 1 KHz (similar to that reported in the Meta paper above). Compared to this, a CMOS based framed camera solution consumes ~250 mW (not compatible with smart glass applications) and data speeds of ~120 Hz. Data speeds are important in eye tracking since it provides finer resolution, important for various psychological aspects like emotional state, intent and interest.

Mr. Verra is excited about the collaboration with Tobii: “Event-based vision is a perfect match for the growing demand for low-power, always-on sensing in next-generation wearable devices. This partnership builds on a strong foundation of collaboration between our teams and expands our joint capabilities into the eye-tracking space. With Tobii’s proven leadership in eye tracking, combined with our experience in event-based sensing, we aim to set a new standard for what’s possible in AR/VR and smart eyewear.”

Tobii, founded ~ 20 years ago, is headquartered in Sweden, and bills itself as a leader in eye tracking (delivered the world’s first remote eye tracker). It serves thousands of enterprises, universities, and research institutes around the world, and is a pioneer in attention computing (Figure 2) which it defines as “technology that understands human attention and intent”. Listed on the Swedish NASDAQ, it has 600 employees and annual revenues of ~$90M.

Tobii uses machine learning, artificial intelligence, and advanced signal processing to decode head and eye movements, translate microscopic gestures into accurate gaze signals, and generate insights to reveal what captures a person’s attention and interpret intent. Applications fall into four broad verticals:

1) Consumer research and user experience: transitions the market research process from a formal survey-based one (in which questions posed to respondents invariably create bias) to a emotional one in which subtle cues gained from eye tracking like gaze, pose and presence can provide valuable information to modify products and packaging. The data can be gathered in formal or informal settings (like when customers shop and are looking at alternate products).

2) Developmental psychology and healthcare research: Eye tracking data offers valuable insights into human behavior and cognitive processes, making it a versatile technology across a wide range of research domains like psychology, medical diagnostics and education.

3) Consumer technologies in gaming and wearables: System-on-chip solutions for eye tracking provide seamless interactions between users and their devices in wearables, smart glasses and AR/VR headsets. Sony is one of Tobii’s customers (Sony PlayStation product lines).

4) Automotive, driver monitoring systems (DMS): Tobii Autosense enables OEMs to build for tomorrow, pushing the boundaries of what’s possible, and achieve true in-cabin differentiation and safety monitoring.

Tobii’s hardware comprises a suite of wearable eye trackers. Software integration and ensuring that the devices work robustly across all populations (age, skin color, eye color, illumination conditions, vibrations, facial and eye-makeup, etc.) are critical challenges. To date, the company has sold a few million units. According to CEO Anand Srivasta, the eye tracking solutions market is in its infancy. He expects this number to grow as the adoption of this technology accelerates, especially in the consumer wearables, industrial productivity and DMS markets.

Event based sensing can deliver significant disruptions in certain verticals where low size, weight, power consumption and heat generation are critical (wearables) and high data rates can enhance the eye tracking resolution to provide more insights into consumer intent and emotion. Other applications may continue to use framed CMOS cameras (when full images are needed, like in DMS). He expects the collaboration with Prophesee to“complement to our existing camera technologies, giving our customers even more options and flexibility when designing their products. Smart eyewear is one of the most demanding segments for eye tracking—requiring ultra-low power, high performance, and seamless integration into a standard glasses form factor.”

As technology accelerates, data generation volume increases exponentially. In many cases, most of this data is useless, non-eventful and non-actionable. Attempting to analyze all of it slows down decision making and insights, and risks missing events that are important. Event based sensing has a significant role to play in lowering compute, storage and thermal management costs, providing higher actionable data rates and opening up new markets and technological acceptance.