I tried the iPhone 15 Pro’s new spatial video feature, and it will be the Vision Pro’s killer app

I’ve had exactly two Apple Visio Pro experiences: one six months ago, on the day Apple announced its mixed reality headset, and the other just a few hours ago. And where with the first experience I felt like I was swimming across the surface of the headset’s capabilities, today I feel like I’m qualified as a Vision Pro diver. I mean, how else am I expected to feel after not only experiencing spatial video on the Vision Pro, but also shooting this form of video for the headset with a standard iPhone 15 Pro?

By now, you probably know that iOS 17.2, which Apple released today as a public beta, will be the first time most of us will gain experience with spatial video. Granted, initially it will only be half the experience. Your iPhone 15 Pro and iPhone 15 Pro Max will, with the iOS 17 update, add a new videography option that you can toggle under Camera Formats in Settings. Once the Vision Pro ships, sometime next year, the format will turn on automatically for Vision Pro owners who have connected the mixed reality device to their iCloud accounts.

I got a sneak peek at not only the new iPhone 15 Pro capabilities, but at what the life-like content looks like viewed on a $ 3,499 Apple Vision Pro headset – and I now realize that spatial video could be the Vision Pro’s killer app.

A critical iPhone design tweak

Apple Vision Pro spatial video

(Image credit: Apple)

To understand how Apple has been playing the long game with its product development, you need only look at your iPhone 15 Pro or iPhone 15 Pro Max, where you’ll find a subtle design and functional change that you likely missed, but which is obviously all about the still unreleased Vision Pro. It turns out Apple designed the iPhone 15 Pro and Pro Max with the Vision Pro's spatial needs in mind, taking the 13mm ultrawide camera and moving it from its position (on the iPhone 14 Pro) diagonally opposite the 48MP main camera to the spot vertically below with the main camera, which on the 14 Pro was occupied by the telephoto camera; the telephoto camera moves to the ultrawide's old slot.

By repositioning these two lenses, Apple makes it possible to shoot stereoscopic or spatial video, but only when you hold the iPhone 15 Pro and iPhone 15 Pro Max in landscape mode.

It is not, I learned, just a matter of recording video through both lenses at once and shooting slightly different angles of the same scene to create the virtual 3D effect. Since the 13mm ultrawide camera shoots a much larger frame, Apple’s computational photography must crop and scale the ultrawide video to match the frames coming from the main camera.

To simplify matters, Apple is only capturing two 1080p/30fps video streams in HEVC (high-efficiency video coding) format. Owing to the dual stream, the file size is a bit larger, creating a 130MB file for about one minute of video.

Even though these spatial files are ostensibly a new media format, they will appear like any other 2D video file on your iPhone or Mac. However, there will be limits. You can trim one of these videos, but you can’t apply any other edits, lest you break the perfect synchronization between the two streams.

The shoot

Apple Vision Pro spatial video

Spatial video capture arrives on the iPhone 15 Pro and 15 Pro Max with the iOS 17.2 public beta update, which anyone can download today (you have to change your settings to accept beta updates). Note that you’ll only be shooting horizontal spatial video (Image credit: Apple)

For my test, I used a standard iPhone 15 Pro running the iOS 17 developer beta. We had already enabled Spatial Video for Apple Vision Pro under Settings in Camera/Formats. In the camera app's video capture mode, I could select a tiny icon that, naturally, looks just like the Vision Pro to shoot in Spatial Video mode.

When I selected that, the phone guided me to rotate the phone 90 degrees so it was in landscape orientation (the Vision Pro icon rotates as soon as you tap it). I also noticed that the image level tool, which is optional for all formats, is on by default when you use spatial video. This is because spatial videos are best when shot level. In fact, shooting them in situations where you know you might not be able to keep the phone level, like an action shot, could be a bad idea. Mostly this is about what it will feel like to watch the finished product in the Vision Pro headset – lots of movement in a 3D video a few centimeters from your face might induce discomfort.

Similarly, I found that it’s best to keep between three and eight feet from your subject, so they don’t end up appearing like giants in the final spatial video.

I shot a couple of short spatial videos of a woman preparing sushi. I tried to put the sushi in the foreground and her in the background to give the scene some depth. Nothing about shooting the video felt different from any others I’ve shot, though I probably overthought it a bit as I was trying to create a pair of interesting spatial videos.

Even though the iPhone is jumping through a bunch of computational hoops to create Spatial Video out of what you shoot, you should be able to play the video back instantly. We handed over our phones and then, a few minutes later, we were ready to view our videos in the Vision Pro.

Hello, my old friend

Apple Vision Pro spatial video

(Image credit: Apple)

While I was worried that after all these months, I wouldn’t remember how to use the Vision Pro, it really only took me a moment or two to reorient myself to its collection of gaze, gesture, and Digital Crown-based controls. It remains a stunningly intuitive piece of bleeding-edge tech. I still needed to hand over my glasses for a prescription measurement so we could make sure Apple inserted the right Zeiss lenses (you don’t wear glasses when using the headset). It’s a reminder that, unlike an iPhone, the Vision Pro will be a somewhat bespoke experience.

For this second wear session, I did not have the optional over-the-head strap, which meant that, for the first time, I felt the full weight of the headgear. I did my best to adjust the headband using a control knob near the back of the headset while being careful not to over-tighten it, but I’m not sure I ever found that sweet spot (note to self: get the extra headband when you do finally get to review one of these headsets).

There were some new controls since I last tried the Vision Pro – for example, I could now resize windows by looking over at the edge of a window and then by virtually pinching and pulling the white curve that appears right below it. I got this on the second try, and then it became second nature.

I finally got a good look at the Vision Pro Photos app, which was easy to navigate using my gaze and finger taps – you pinch and pull with either hand to swipe through photos and galleries. I usually kept my hands in or near my lap when performing these gestures. I looked at photos shot with the iPhone 15 Pro at 24MP and 48 MP. It was fun to zoom into those photos, so they filled my field of view, and then pinch and drag to move around the images and see some of the exquisite detail in, for instance, the lace on a red dress.

I got a look at some incredible panorama shots, including one from Monument Valley in Arizona and another from Iceland, which featured a frozen waterfall, and which virtually wrapped all the way around me. As I noted in my original Vision Pro experience, there’s finally a reason to take panoramic photos with your iPhone.

Head spatial

Apple Vision Pro spatial video

This spatial video scene was one of the most effective. Those bubbles appeared to float right by my face (Image credit: Apple)

Inside the Vision Pro Photos app is a new media category called Spatial. This is where I viewed some canned spatial videos and, finally, the pair of spatial videos I shot on the iPhone 15 Pro. There was the campfire scene I saw during my WWDC 2023 experience, a birthday celebration, an intimate scene of a family camping, another of a family cooking in a kitchen, and, my favorite, a mother and child playing with bubbles.

You can view these spatial videos in a window or full-screen, where the edges blend with either your passthrough view or your immersive environment (a new environment is Joshua Tree) that replaces your real world with a 360-degree wraparound image. In the bubble video, the bubbles appeared to be floating both in the scene and closer to my face; I had the impulse to reach out and touch them.

In the kitchen scene, where the family is sitting around a kitchen island eating and the father is in the background cooking, the 3D effect initially makes the father look like a tiny man. When he turned and moved closer to his family, the odd effect disappeared.

It’s not clear how spatial video shot on iPhone 15 Pro is handling focal points, and if it’s defaulting to a long depth of field or using something different for the 3D effect. You can, by tapping your iPhone's screen during a spatial video shoot, set the focus point but you can't change this in editing.

My two short videos were impressive, if I do say so myself. During the shoot, I did my best to put one piece of sushi the chef held up to me in the foreground, and in the final result, I got exactly the effect I was hoping for. The depth is interesting, and not overbearing or jarring. Instead, the scene looks exactly as I remember it, complete with that lifelike depth. That’s not possible with traditional videography.

What I did not do was stand up and move closer to the spatial videos. Equally, these are not videos you can step into and move around. You're still only grabbing two slightly different videos to create the illusion of depth.

In case you’re wondering, the audio is captured too, and this sounded perfectly normal. I didn't notice any sort of spatial effect, but these videos were not shot with audio sources that spanned the distance of a room.

Apple Vision Pro example

In this sample provided by Apple, you can see how the candle smoke appears to float toward you – it’s a trippier effect when you’re wearing the Vision Pro headset (Image credit: Apple)

What’s next?

Because you’ll have spatial video shooting capabilities when you install the iOS 17.2 public beta you could be shooting a lot of spatial video between now and when Apple finally starts selling the Vision Pro to consumers. These videos will look perfectly normal – but imagine having a library of spatial video to swipe through when you do finally buy the Vision Pro. That, and the fact that your panoramas will look stunning on the device, may finally be the reason you buy Apple's headset.

Naturally, the big stumbling factor here is price. Apple plans on charging $ 3,499 (around £2,800 / AU$ 5,300) for the Vision Pro, not including the head strap accessory, which as mentioned, you’ll probably need. That means that while millions may own iPhone 15 Pros and be able to shoot spatial video, a precious few will be able to watch them on a Vision Pro.

Perhaps Apple will make the Vision Pro part of one of its financing plans, so that people can pay it off with a monthly fee. There might also be discounts if you buy an iPhone 15 Pro. Maybe not. Whatever Apple does, spatial video may make the most compelling case yet for, if not owning a Vision Pro, then at least wishing you did.

You might also like

TechRadar – All the latest technology news

Read More

The Apple Vision Pro’s first 3D movies have just shown up in the App Store

Multiple 3D movies have reportedly appeared on the Apple TV app seemingly in preparation for the launch of the Vision Pro headset next year.

The updated support was discovered by tech news site FlatpanelsHD after digging through the recently released tvOS 17.2 beta. Apparently, there was more to the patch other than introducing a redesigned UI to Apple TV. According to the report, the 3D movies that can be found on the platform include, but are not limited to, Jurassic World Dominion, Pacific Rim Uprising, and Shrek. The full list can be found on FlatpanelsHD, which primarily consists of action films. Each title will have a 3D-compatible icon on their respective details page letting you know of its support.

It’s important to mention that every single title has had a 3D cinema release in the past. There aren’t any original 3D movies or series, at the time of this writing. This leads us to believe that maybe Apple has created a new file format for the Vision Pro. Studio developers could’ve converted the films into said format so they can be played on the headset. However, we don’t know for sure. This is just speculation on our end.

Immersive and comfortable

Obviously, there isn’t a way to actually view these movies in their intended way since the Vision Pro isn’t out yet nor do we know “what resolution and frame rate these 3D movies [will] play in.” Each eye on the headset can output 4K resolution so that’s one possibility. Older titles, like Shrek, will most likely have to be remastered to a higher quality.

Although the resolution remains unknown, we have some idea as to what the experience will be like. Apple has a video on its website teaching developers how to prepare content for visionOS. The 16-minute lesson is pretty complex, but the main takeaway is that Apple is taking care to ensure watching content on the Vision Pro results in an immersive and comfortable experience.

The headset utilizes stereoscopic 3D, a technique where the device creatively uses flat images to produce the illusion of depth. One eye will see one image while the other eye sees a “slightly different perspective”. Overlay the two on top of each other and you get a 3D view.

It’s similar to how our own eyes perceive the world around us as each one sees objects in a slightly different manner. This difference is called parallax and it’s something the tech giant is striving to nail. Rendered elements in a 3D video without parallax can “cause discomfort when viewing.”

Bringing back an old idea

It’ll be interesting to see what else comes from this support. As FlatpanelsHD points out, Apple could invertedly resurrect 3D movies as the new hardware enables the format. Maybe 3D TVs might make a comeback. 

They’ve seemingly gone the way of the dodo. There are, however, a few companies out there eager to revive the old idea like Magnetic3D. Now we just need the content, which could be led by the upcoming Godzilla series, Monarch: Legacy of Monsters if the latest rumors are to be believed. 

While we have you, be sure to check out TechRadar's latest round-up of the best VR headset deals for November 2023.

You might also like

TechRadar – All the latest technology news

Read More

I’d love it if Apple dropped the Vision Pro’s worst feature to make a cheaper version – it’s a win-win

Well, the Apple Vision Pro isn’t even available yet, and Apple is already looking to the future (well, the further future) to consider how to make the follow-up model even better. Or, in this case, even cheaper: according to a new report from Bloomberg’s Mark Gurman, Apple wants its second headset to cost less.

We already knew Apple was likely considering a cheaper Vision Pro model, and this latest news gives credence to those rumors. It seems like the first-generation Vision Pro is having some teething issues too, most notably the weight – the same Gurman report discusses Apple’s efforts to reduce the weight of a second-generation headset. But there’s a far better nugget of information we should be focusing on…

Apple might get rid of that horrible external display! Yes, the EyeSight feature that I heavily criticized when the headset was first revealed might be cut from future Vision models in order to save on costs. It makes a lot of sense; adding an external screen to show the user’s eyes is a ridiculous feature that no doubt costs a lot of money, adding extra (entirely unneeded) OLED display tech to every single headset. The screen even has lenticular glass to create an illusion of depth.

The feature isn’t even that helpful; it projects the eyes of your uncanny-valley FaceTime avatar onto the exterior screen when you’re looking through the headset’s external cameras, or displays an opaque color when you’re immersed in something that blocks out your surroundings. In other words, it tells other people if a Vision Pro user can see them or not – something that could’ve probably been achieved with a single LED.

Get ready for the Apple Vision… something

Apple is reportedly targeting a $ 1,500-$ 2,000 price range for its more affordable Vision headset, which sounds a lot more accessible than the current $ 3,499 price tag. Given that some other popular VR headsets – like the new Meta Quest 3 – are a lot cheaper, many potential users are likely to give the first-gen Vision Pro a pass.

A cheaper model was likely always going to happen given the name Apple chose for its headset – Vision Pro implies the (future) existence of a standard Apple Vision headset, in keeping with Apple’s naming conventions for its other products. Just look at the iPhone 15 Pro and the MacBook Pro – hey, maybe the cheaper headset will be called the Apple Vision Air?

Apple is also reportedly developing a second high-end version of the headset (a Vision Pro Max, perhaps?), and it’s enough to make me wonder if the first-gen model will even be worth buying at all. The Vision Pro represents Apple’s first step into an entirely new market, and it wouldn’t even surprise me if it gets hit with delays – after all, it’s suspected that issues with chip manufacturer TSMC could delay M3 MacBook Air models until the middle of next year, and the Vision Pro features not just the M2 chip but also a new dedicated R1 chip for mixed-reality workloads.

In any case, I’ll be happy to see a less expensive headset from Apple. I still remember the noise the crowd made at WWDC 2023 when the price was unveiled… and let’s face it, EyeSight is a feature nobody really needs.

You might also like

TechRadar – All the latest technology news

Read More

Apple Vision Pro’s killer app could be… Windows XP?!

When the Apple Vision Pro, the powerful and ultra-expensive VR headset, was shown off by CEO Tim Cook at WWDC 2023, few (if any) of us expected one of the most interesting showcases for the new tech to be running the ability to run a Windows operating system by arch nemesis Microsoft – especially one that was first released back in 2001. But you know what? It actually is.

As 9to5Mac reports (via iPhoneSoft), developers working on an early version of the visionOS operating system that the Vision Pro will run on, have managed to get an emulator running with a working version of Windows XP.

In a video posted on X, the social media network formerly known as Twitter, which you can see below, Windows XP is shown loading in a big floating window in a lounge.

See more

Room with a view

While Windows XP is regarded as one of the better versions of Microsoft’s operating system, the idea of it looming over you as you sit on your couch may seem like some sort of dystopian nightmare, but this is actually pretty cool.

Sure, it’s unlikely that I’d want to fire up Windows XP to play some Minesweeper on a virtual 100-inch screen, this is an exciting demonstration on what could be possible for Vision Pro.

The developers are working on UTM, an emulator that brings non-Apple software to iPhones, Macs and now, it seems, the Vision Pro.

This emulation isn’t yet perfect – there isn’t a way to control Windows XP when it’s running – but the team has time to work on that before the Vision Pro’s official launch early next year.

And, while Windows XP is shown off in this video, it does suggest that this could mean other operating systems could come to Vision Pro. This would open up huge possibilities, as you’d be able to run full programs and games on the headset.

Apple Vision Pro

(Image credit: Apple)

Killer app? Perhaps

One of the major questions many people – including myself – had when Apple showed off the Vision Pro, was what is the headset actually for? We were shown some concept videos of people making video calls and watching movies using the headset, but nothing that really justified the huge $ 3,499 (around £2,815 / AU$ 5,290) price tag that it will launch with.

What we need is a killer app that makes the Vision Pro a must-by. So far, we’ve not had that, but an app that allows an almost unlimited amount of applications could be the key – and would also showcase Apple’s vision for ‘spatial computing’, which is how the company refers to the tech powering the Vision Pro – which includes hardware such as the same M2 chip found in the best MacBooks, and the new R1 chip.

The UTM app is certainly exciting, but I wouldn’t get too excited just yet. As you may imagine, Apple won’t be too keen on people running non-Apple software on the Vision Pro, so don’t expect installing UTM on the headset to be as straightforward as downloading it from the built-in app store.

It would be a shame if Apple hobbled the Vision Pro’s potential by forcing a walled-garden approach to apps, like on the iPhone, where you can only officially install apps from Apple’s own store, unlike the more open approach on Macs and Windows laptops

If Apple is serious about the Vision Pro being a productivity machine and the dawn of ‘spatial computing’, then it’s going to have to be willing to give up some control – and it may not want to do that.

TechRadar – All the latest technology news

Read More

visionOS: everything you need to know about the Vision Pro’s operating system

After over two years of rumors and speculation, Apple has finally revealed its upcoming VR headset, the Vision Pro, at this year’s WWDC 2023 event. The Vision Pro is nothing if not impressive, with capabilities like outputting 4K resolution from each stamp-sized display and creating a “digital persona” just by scanning your face.

But as Mike Rockwell, VP of the Technology Development Group at Apple, states in the announcement “none of [the] advanced technology could come to life without… visionOS.” It’s described as the first operating system specifically designed for “spatial computing”. 

visionOS is described as the first operating system specifically designed for “spatial computing”. And it built on the same building blocks as macOS and iOS, but comes with unique features to better facilitate virtual reality. 

visionOS building blocks

(Image credit: Apple)

For example, visionOS comes with a Foveated Renderer, similar to the PSVR 2. What it does is increase the visual fidelity of whatever a person is looking at while blurring everything in your peripheral vision. 

Rockwell then goes on to describe the operating system’s “multi-app 3D engine” allowing “different apps to run simultaneously”.

In the workplace

As impressive (or long-winded) as it all sounds, you may be wondering what it all looks like? 

In its most basic form, visionOS looks pretty much like any other virtual reality platform. You have large windows floating in the space in front of you. Turning your head lets you see switch tabs so you can go from Safari to Messages. And when you launch the Vision Pro, you get an assortment of apps to choose from. Pretty simple stuff. 

Where visionOS truly shines is in its individual use cases. You will be able to send 3D image files via Messages as well as display that model right in front of you in every possible angle. The system also responds to the natural light around you so the 3D objects will have shadows befitting of the environment you’re in. It can help you understand scale as well as distance. Professionals can create their own setup for work by arranging apps to their liking. 

Control in visionOS will, for the most part, be done with your hands, eyes, and voice; however people will be able to connect Bluetooth peripherals like the Magic Keyboard if they prefer a more physical interaction.

Man working with Vision Pro

(Image credit: Apple)

Beyond first-party software, visionOS will be running third-party apps natively at launch. This includes the likes of Adobe Lightroom, Microsoft Teams, and Zoom. It’s unknown if other apps like Photoshop will be present upon release.

Additionally, the operating system is set to run specific education apps. The one that impressed us the most is a piece of medical software for looking at renders of the human body. The announcement video shows an exploded view of the human heart, complete with ventricles as well as the pulmonary artery. There’s an engineering app to help people visualize certain physics phenomena like air flowing over a race car. 

On the surface, it sounds like Apple is launching its own version of the Microsoft Hololens with all work-centric tools. But it's a lot more nuanced, as the tech giant is going for broke by incorporating some impressive entertainment features. 

air flowing over race car in visionOS

(Image credit: Apple)

And at home

At launch, Disney Plus will be available alongside over 100 games via the Apple Arcade platform. You can expand the VR screen to massive proportions like you’re in a movie theater. 3D movies will be supported too, so if you want to watch Avatar: The Way of Water as it was intended, the option does exist. Users can have the screen floating in their room, but if they want something more dynamic, the background can be replaced with a different environment like deep space or Mt Hood in Oregon.

We do wish Apple showcased more of the gaming side of things. We saw it’s possible to expand a game screen to a larger size so you can get a better view. Gamers won’t have to play with their hands as visionOS will support gamepads, namely the PS5 DualSense controller. Hopefully, support will extend to other peripherals like the Nintendo Switch's JoyCons.

Man watching movie in visionOS

(Image credit: Apple)

Apple didn't have much in the way of specific titles. If anything, the company seemed more interested in having its headset and operating system adhere more towards providing an experience rather than being a bonafide gaming computer. The keynote revealed a Star Wars VR adventure where you fly around the universe of The Mandalorian. However, there was nothing in terms of a lightsaber duel or anything action packed.

That’s pretty much everything there is to know about visionOS, at least to how it pertains to the everyday user. The rest mostly relates to software development. At the end of the keynote, Apple revealed the operating system will support the Unity game engine. This could mean the company is paving the way for developers to come in and create video games for visionOS. However, given Apple’s spotty history with video games, it remains to be seen if this will attract any developers at all.

WWDC 2023 recently concluded and a ton was shown off from MacOS Sonoma to a brand new Mac Pro. Be sure to check out TechRadar’s coverage of the event

TechRadar – All the latest technology news

Read More