The Future of AI Is Coming for Our Eyes: VR and AR’s Killer App, With Killer Unknowns

I was immersed in a world of virtual things: 3D maps of my home and places I’d never been, videos wrapping around my head, and browser pages floating all around me. But I wasn’t alone. A friendly voice was there, listening to my questions and understanding what I was seeing. This companion seemed to see what I saw and hear what I heard. Was Google’s Gemini AI behind me, surrounding me or inside me? Where did my perceptions end and AI’s begin?

I was demoing a future Samsung mixed-reality headset with Google’s Gemini AI 2.0 inside it. The headset won’t be out until later in 2025, but it’s as good a sign as any — not to mention a warning — of what’s coming soon in personal tech. AI has listened to and responded to us for years. It’s heard our voice prompts, read our text prompts and scanned our photos, all through our laptops, phones and in the cloud. Next up, AI has its sights set on our eyes.

These ideas aren’t new, but we’re on the verge of seeing companies flip the switch, making surprising things happen with headsets and glasses — some already available, others still on the horizon. Google’s Android XR chess move is just the first. Expect Meta, Apple, Microsoft and plenty of others to follow right along. Some already are. From what I’ve already seen, it’ll make what we think about AI now seem like the opening act.

Google’s Android Ecosystem President Sameer Samat sees AI and XR (the industry’s current abbreviation for “extended reality,” a space that covers VR, AR and AI-assisted wearables) becoming a natural fit. “It can actually help you control the UI. It can work on a problem collaboratively with you and take actions in that virtual space with you,” Samat told me. My demos in Android XR offered glimpses of this, showcasing an AI companion experience unlike anything I’ve tried before. It felt more personal, as if the AI was almost living in my head, seeing what I saw.

That future is already here. Meta’s updated Ray-Bans now include live AI assistance and translation, all in a pair of $300 glasses you can buy today.

Entanglements and opportunities

For the last few years, AI has become an overwhelmingly hyped part of tech, driven mostly by successes in generative AI by companies like OpenAI. AI’s magic tricks are sometimes amazing, sometimes disappointing, sometimes promising and sometimes garbage. As with many overhyped technologies, promise and reality often intertwine, leading to chaos and disruption before the true impact can be understood.

I find the whole AI landscape confusing, even after working in tech journalism for years. I don’t know how much I find it useful or horrible. Sometimes it’s both, but I often think about complexity and acceleration. When new technologies gain traction and go mainstream, the outcomes can be unexpected — just as they were with phones. Most people don’t wear VR and AR headsets and glasses right now, or if they do, it’s not often. But that could change, and as AI is able to collect data through an increasing number of sensors on our faces, the possibilities — if things do go large scale — are hard to comprehend.

I’ve seen snippets of potential. My Samsung and Google demos showed me how to ask my glasses or headset to be my memory and recall things I’d seen. I could ask for information and clarification about anything I was doing, like having a living search engine at my side. This year, I’ve been wearing Meta’s Ray-Ban glasses — ordinary-looking glasses available since last fall. Now, they can identify objects or translate languages instantly with a voice command and a quick shutter snap. I’ve wandered through my neighborhood, asking my glasses about the things I see. Sometimes they’re helpful; other times, not so much.

I’m spending more time in headsets and glasses. Yes, it’s for my job as an early-tech explorer. But VR used to be about trying novel experiences and games, but now it’s become part of my routine. I use a Quest headset for weekly workouts with a virtual trainer, tracking my heart rate as I go. I slip on a Vision Pro to work, stretching a curved display around me, immersing myself in music, sitting on the moon and taking breaks to watch floating movies. I take walks with glasses that play meditations and music, make phone calls for me and capture little memories of my life. Technology is increasingly becoming a part of my daily life, right in front of my eyes, and now AI is poised to join me on that journey. What happens next?

AR and AI as a brain-computer interface?

Some companies, like Meta, are beginning to explore neural input devices, as I experienced with its prototype Orion glasses. Small wristbands can detect electrical signals using EMG (electromyography), turning those signals into predictive gestures. AI already works all over the place on VR and AR headsets to predict head movement, track eye movement, turn hand gestures into actions and sync experiences to make them feel realistic and not nauseating. But more advanced generative AI assistants could also start to make headsets feel like the closest thing we have, short of implants, to a brain-computer interface.

Some companies I’ve visited, like OpenBCI, are already exploring combinations of EEG sensors and VR/AR. But AI working with visual and audio cues, along with hand movements and gestures, could do enough to feel like mindreading, too. Eye tracking is already a field full of possibilities and risks as far as how indicative eye gaze can be of our thoughts and cognitive state. 

It’s hard to wrestle with complexity, but I keep thinking about Ray Kurzweil. The famous (and sometimes controversial) AI pioneer, now a director of engineering at Google, has written about the rise of AI for decades. His 2004 book, The Singularity Is Near, explored a strange future shaped by accelerating AI developments, supported by pages of charts and graphs. In 2024, Kurzweil released The Singularity Is Nearer, a 20-year follow-up that offers a more concise revisit of his earlier arguments. What made me take notice was realizing how many of Kurzweil’s thoughts on AI have come to pass since his last book. His future predictions range from the bizarre to the unbelievable, including nanobots that rejuvenate us, solutions to energy crises, and the end of economic disparity. However, Kurzweil envisions a bridge between the singularity he predicts and our present moment — and he believes it lies in AR and VR.

In a conversation with Kurzweil earlier this year, he told me as much. “Yeah, it’s much better than just trying to control a phone. In an AR environment, things could be presented to you, you could absorb them much more quickly. And it’s better than going into your brain. It’s easier, and I think that will be the next step. I do think, ultimately, we’re going to want to extend our brains into the cloud. But AR, I think it’s a step between where we are today and where we’ll end up.”

Mixed-reality headsets and smart glasses are nowhere near being able to directly interface with our brains, but generative AI connected to increasingly-activated cameras and microphones starts to feel like a step toward that vision. In 2025, we’re likely to see many new experiments pushing the boundaries of this approach.

Lots of players in the landscape, lots of ambition

Google’s the latest mover aiming to layer AI into XR, but Meta’s already been exploring this field. Meta’s CTO, Andrew Bosworth, told me a year ago that AI was going to be added more into both Meta’s glasses and Quest VR headsets over time. Michael Abrash, chief scientist at Meta Reality Labs, has long envisioned AR glasses as assisted memory systems and agent-based AI interfaces. Meta’s Orion prototype glasses demonstrated parts of this during a recent demo I experienced, and the latest Ray-Bans are introducing reminders and continuous assistance through an always-active, recording camera.

Generative AI in VR, meanwhile, has focused on creative tools — for now, at least. “We’re starting with gen AI in Horizon [Quest headsets] for world building and for your own identity and customization, and avatar and clothes and accessories, and for animating these characters,” Meta’s Mark Rabkin, head of the Horizon platform, told me at the company’s Connect developer conference earlier this year. But Rabkin sees a necessary layer of visually aware AI inside VR and AR, much like Android XR’s Gemini, being a key next step. “Pretty much everything you do with Ray-Bans you can eventually do in the metaverse. But for it to work, Meta AI needs to tell you about the metaverse.”

Bosworth mentioned to me in a more recent conversation that training data for AI’s recognition of virtual things still isn’t great. It’s better at recognizing the real world based on training from photos and videos, something that camera-based glasses can do better. As Meta’s glasses shift from voice-based glasses to something with a display, that could also introduce hand tracking and wristband-like accessories.

Apple now has its own bleeding-edge mixed-reality headset, but the camera-studded Vision Pro doesn’t have a deeply aware layer of generative AI baked in… yet. Apple is layering bits and pieces of generative AI via Apple Intelligence, announced back in June, into its phones, iPads and Macs. The Apple Vision Pro, an early-adopter headset, hasn’t gotten Apple Intelligence, but it’ll likely be next on deck.

There are already signs of how Apple’s mixed-reality AI might work. Visual intelligence just debuted on the iPhone, scanning and identifying things in the world with a press of a side button much like Google’s Lens feature. Apple’s expected to make a more affordable version of the Vision Pro as early as next year, possibly connecting to the iPhone for the first time. It would make a lot of sense to start adding more assistive camera-based AI features onboard then, if Apple’s ready for it.

There are lots of other players, too. Snap introduced its own developer-focused standalone Spectacles AR glasses this past fall that I also tried out, and already have some ChatGPT-integrated generative-AI capabilities. Xreal’s newest glasses have optional cameras specifically for this type of future AI functionality. 

Almost all AR and VR headsets now have vast arrays of higher-quality cameras onboard, which are already used to blend video feeds of the real world and virtual overlays to create mixed reality. Add a deeper layer of AI, and these sensors could be a way of creating continuous agent-like awareness on tap. These layers of AI might also change the way apps and games are made. Instead of staying in one experience for a while, it’s more likely that future headsets will keep mixing several experiences at once, while AI helps manage it all.

Phones will start becoming more directly connected with these new headsets and glasses, too. Much like what Android XR is already hinting at, expect the way we manage these extra services on headsets will be an extension of the phones we already use.

How will we draw the lines on privacy?

AI already scans our words, our voices and our photos. AI integrated into headsets and glasses is a glimpse into a world where their cameras will scan our entire lives, or at least everything we see. It’s an unsettling thought. To make this work, the cameras need to have access to AI, and companies need to work out the permissions and privacy features for it all to not feel intrusive or invasive.

In my Android XR demos, it felt like Gemini could see everything I was doing in the headset, but it also felt like it could see everything around me in my room. Microsoft’s Recall feature on its Windows PCs faced backlash and concerns that its always-on awareness of your computer activities could see private data and expose moments people don’t want shared or known by an AI service. This concern over privacy has held back companies like Meta and Apple from turning camera access on for developers in its mixed reality headsets. But those barriers are falling. Meta’s opening up camera access and Apple’s doing it, too — at least for businesses to start. 

Meta’s Ray-Bans can identify lots of things, but with limits. I’m often not allowed to identify a car or a particular location or address — Meta AI says it’s against the privacy terms. Sometimes, though, with the right prompt, I could. I can’t ask about health or nutritional information in products. And in my recent Android XR demo with Gemini, I wasn’t able to try to identify my colleague’s face.

Where will these AI memories feel like extensions of our own, and where will they put up guard rails — either for our privacy, or for the legal protections of the company making the AI?

We already have phones that can use cameras to link up with AI in all sorts of ways. Whatever limits seem to be imposed by one app or OS can sometimes be bypassed with another. 

And yet, for future headsets and glasses to feel truly assistive, truly aware of the world, they’ll need to link AI with cameras and other sensors even more deeply. What we see emerging in 2025 will likely only scratch the surface, but the potential — for good, bad and utterly weird and messy — are coming. It’s hard to imagine what the implications truly are.

“Whatever that data is you’re streaming through your glasses, your phone, your accounts and services, you care a lot about that. You want to make sure that whoever has that data is trusted, ideally they have as little of it as possible, it’s as local as possible, it’s held on the server as limited as possible. And on the flip side, you really want an AI that learns you, specifically,” Meta’s CTO, Andrew Bosworth, told me over Zoom as the year ended. “I think we’re going to benefit a lot from personalized AI. There’s no reason our personalized AI has to come at the cost of privacy.” 

But Bosworth sees continual access to your life, in AI, as part of what’s coming next — in glasses, on headsets and everywhere. “I think this is a thing we’ll get pretty comfortable with as a society. I’m pretty sure the consumer demand is going to be very high for that.”

Meanwhile, there are plenty of people who raise warning flags. Ed Zitron, host of the Better Offline podcast and a longtime critic of AI’s hype cycle, said: “Generative AI is far less of a privacy problem when it’s user-facing, but the problems come when it can see the rest of the world. The applications that see and process the real world must be regulated and fast, otherwise we’ll see some of the most egregious violations of privacy in history, proliferating the worst of surveillance capitalism at the scale of a social network.”

As I took a stroll in New York testing Meta’s latest live AI update to its glasses, which can continuously record video and observe the real world as I wander, it never felt more clear that things are changing fast. And right now, I’m as amazed and confused and concerned about it all as anyone else.