Apple starts connecting the dots for its next big thing

Nearly 15 years after the iPhone sparked the smartphone revolution, Apple is assembling the pieces for what it hopes will become its next business-altering device: a headset that blends the digital world with the real one.

The company has enlisted Hollywood directors such as Jon Favreau to develop video content for a headset that it is expected to ship next year, according to three people familiar with that work. Favreau, an executive producer of “Prehistoric Planet” on Apple TV+, is working to bring that show’s dinosaurs to life on the headset, which looks like a pair of ski goggles and aims to offer virtual- and augmented-reality experiences, these people said.

Separately, at its annual conference for developers this Monday, Apple plans to unveil software tools that would allow apps to add new camera and voice functionality, laying the groundwork for a hands-free interface that customers will eventually be able to navigate on the headset, according to two people familiar with the project and documents reviewed by The New York Times.

An Apple spokesperson, Trudy Muller, declined to comment on future projects. A spokesperson for Favreau could not comment.

The planned headset will thrust Apple into an emerging competition to define the future of mixed reality. Microsoft, Google and Facebook’s parent company, Meta, are in various stages of developing software and hardware to create environments where 3D digital images and the physical world coexist.

Last year, Meta CEO Mark Zuckerberg dropped Facebook as his company’s name and committed it to building a concept known as the metaverse, where the online, virtual and real worlds coalesce in a new universe. He and others believe it could become the next wave of computing, succeeding the smartphone age dominated by Apple’s iOS and Google’s Android software, much as those platforms followed the decades of Windows and Macintosh.

“This is the next frontier,” said Carolina Milanesi, a technology analyst with Creative Strategies, a technology research firm. “For Apple, this is about a new computing experience and opportunity to engage consumers with a device and new experiences that build on what they’ve done with content.”

The mixed-reality work is expected to be overshadowed during the keynote of Apple’s conference Monday, which will largely be devoted to the company’s existing software systems. Apple may also announce a redesigned MacBook Air with slimmer borders around its display than current models and updated processors, analysts said.

Apple’s development of virtual-reality content and software tools is central to creating experiences that give its future headset purpose. Its last major new product, the Apple Watch, was launched with about 3,000 apps but struggled to take off because tech reviewers said few of those apps were useful. Similar shortcomings have dogged Meta’s Quest virtual-reality headset, which surpassed 10 million sales last year, because many view it as a gaming device.

From its original Macintosh to its iPad, Apple has pursued products that attract a broad swath of potential customers and have an array of uses. It sold an estimated 240 million iPhones last year, accounting for about half of its $366 billion in total sales. To make the headset worthwhile, analysts said, it will need to have utilities that transcend the niche world of video games.

Apple CEO Tim Cook has been talking about the potential of augmented reality for years. In 2016, he told investors that the company was investing heavily in it and considered it a “great commercial opportunity.” Around that time, many employees on Apple’s campus were reading “Ready Player One,” a futuristic novel about virtual reality, and talking about the possibilities of creating Apple’s own mixed-reality world.

Apple hired an engineer from Dolby Technologies, Mike Rockwell, and tasked him with leading the effort. His early efforts to create an augmented-reality product were hobbled by weak computing power, two people familiar with the project said. Continuing challenges with its battery power have forced Apple to postpone its release until next year, those people said.

The augmented-reality initiative has been divisive inside Apple. At least two members of its industrial design team said they had left the company, in part, because they had some concerns about developing a product that might change the way people interact with one another. Such sensitivities have increased inside the company amid rising public concern about children’s screen time.

With Rockwell at the helm, the product would be one of the first to come out of Apple led by its engineering team rather than its co-founder Steve Jobs, who died in 2011, and its former design chief, Jony Ive, who left the company in 2019. The Apple Watch project was led by Ive and his designers, who defined how it looked, operated and was marketed.

Favreau's programming shows how Apple is trying to differentiate its product from Meta’s. It also illustrates how the company is tapping into the relationships it has cultivated in Hollywood since starting Apple TV+ in 2019.

“An amazing headset could give you a better experience than 80-inch TV,” said Matt Miesnieks, CEO of LivingCities.xyz, a startup working on metaverse technology.

The software tools from Apple extend a multiyear push to encourage developers to build augmented-reality apps. The company kicked off that effort in 2017 with ARKit, which allowed developers to use the iPhone’s camera and motion sensors to place digital objects in the real world and let people interact with them.

However, about 70% of Apple’s developers said they did not use that tool, according to a survey of more than 500 developers by Creative Strategies.

A toolkit that Apple is expected to introduce at the conference offers software developers new abilities to trigger shortcuts inside their apps using Siri and QR codes, interactions that will be leveraged in future headsets, a person familiar with the project said.