Pitfalls in Safari Front-End and UI Design for Apple Vision Pro Revealed

In January and February of 2024, the highly anticipated Apple Vision Pro was unveiled to the world, revealing its cutting-edge Safari frontend and UI design. This product has brought a storm to the global tech industry, and the design of this product is like a rainbow after a storm, illuminating the world of technology.

There are 3 key points that have been heavily discussed:

1. Despite the lack of Chinese input, the app still needs to be designed with Apple's aesthetics in mind.
2. Some system UI are only displayed when user isn't moving, which makes it impossible to use while on the move. The window stays in place and needs to be held with your hand in order to move it. However, when the window is being moved, the one held becomes partially transparent, and if moved too quickly, it may fail to be positioned correctly. So, when you see a short video of someone walking around outdoors while waving the window, it's just for show...
3. Highly interactive interfaces, directly manipulated interfaces allow for more intuitive interactions – the 'outside field' mentioned provides a compelling, immersive experience that highlights the products in a dynamic environment.

In each unboxing experience video, it is inevitable that there will be several scenes of using Safari to surf the Internet.
But as a person who is sensitive to screen size when writing UI for web pages, it always looks a little strange...
Why does this Safari window look so small?

About 9 minutes and about 12 minutes, each has a section using the Safari browser

MKBHD, around 14 minutes in, there is a scene of using Safari to browse YouTube.

So I also installed the developer beta version to explore how to use Safari in visionOS?
Note: This requires Xcode 15.1 or above, which requires macOS Sonoma to install, I upgraded from Ventura for this.

Will the 3D objects that used to be gimmicks on websites become more appealing on Vision Pro?

In 2023, on the iPhone 15 Pro, a new spatial video capture technology makes life-like possible on the Apple Vision Pro, making it literally a window to new dimensions!
But how do these spatial videos appear on website? Is there a difference in presentation quality compared to complete environment simulation through 360-degree view displays, model rendering, and more presented in the Apple Vision Pro for an even more immersive experience? Let's delve deeper into this fascinating capability.

360 Degree Photo

You can download some panoramic photos from unsplash for testing. However, when placed on the website, there is still no response, it's just a picture. Web developers still need to design a panoramic image browsing interface.

But if you download this wide jpg image back to your local machine and open it from the file manager, Apple Vision Pro will have a 'Immersive' button in the corner. After pressing it...

This jpg photo seamlessly converts into a 360-degree photo experience, enveloping the viewer.
So, in the future, can we take everything we want to display as 360-degree panoramic photos, put them all in Apple Vision Pro, and ‘show’ them to customers?
Let’s solve the glasses prescription issue first…

Using .usdz and .reality to Display AR Models

Every year after Apple releases new products, there are Apple fans who go to the official Apple website, click on the AR display function at the bottom of the product page, place the AR Apple new products in their own life scenes to appreciate, and they can also take screenshots and post them on social networking sites to show off.

This manifests, for instance, in the creation of a 3D model of a lamp, available in usdz format, which site developers can easily incorporate into a website. This technology not only surpasses traditional high-definition expectations but, when interacted with through iPhone, iPad, and other AR-compatible apps, provides an extraordinary depth to the user experience, a feature not yet accessible to Android device holders.

In Apple Vision Pro, this feature is surprisingly available and even more powerful. The same usdz file can be repeatedly placed N times into the 'space,' just like a virtual exhibition. The image below shows several shoes placed at different angles and sizes.

Apple AR supports not only usdz but also reality files, which include a 3D interactive model with sound effects and descriptions. If you're interested, you can check out Augmented Reality – Apple Developer.
Below is a smaller model that runs more smoothly, giving you an idea of the experience:

Although these features can also be used on mobile phones and tablets, in terms of real-time interaction and intuitiveness, Apple Vision Pro is definitely better and more direct. It seems that some organizations have already started to make purpose, not only investing heavily in various 3D unit teaching materials but also applying for funds to purchase Apple Vision Pro for research purposes.

Virtual Exhibition

The principle and production method are basically similar to the 360-degree panoramic photos mentioned above, except that some parts are man-made and not entirely shot using 360 panoramic cameras.
During the pandemic, this became very popular. Taking Taiwan's experience as an example, it includes things like the following:

If you cannot see the iframe, please visit Taiwantrade Hosiery Exhibition

In simple terms, users can drag to adjust the angle, and usually, clicking on certain parts will display on-site explanation text or link to the explanation.
On Apple Vision Pro, there are no additional special effects as mentioned earlier. This is because Safari has limitations on window size and does not support the HTML5 Fullscreen API, which is quite unfortunate.

Can spatial videos be placed on a webpage?

Spatial videos are exactly what they sound like, videos that can only be captured by iPhone 15 Pro and Apple Vision Pro. You can find some examples online:
– How to View Vision Pro’s Spatial Video Before You Get Your Headset includes Spatial Video Samples
– Some Spatial Concert Videos for Ya’ll by pashdown – Reddit

After downloading these mov or mp4 videos, you can directly open them in Apple Vision Pro. There will be a small button in the upper right corner labeled 'Immersive Mode,' and the video will appear as if it's a magical portal in the space (as shown below):

When viewed in a more immersive angle, the scene will appear three-dimensional, and there will be spatial audio! The feeling is somewhat like watching a Harry Potter movie, but this 3D effect is just a glimpse of the VR photo experience, not a full-fledged immersive view.

However, if you embed the spatial video mov or mp4 in a webpage using the HTML5 video tag, it will just be an ordinary video. Others won't be able to tell this is a spatial video, even if you enlarge it by clicking the upper left corner, it will just hide the browser frame (as shown below):

Updated on 2024/2/13:
Competitor Meta Quest also mentioned in their official statement that they support spatial videos. You can see the demonstration around 5:10 in the video Spatial video on Meta Quest.

Meta Quest 3 Vs Apple Vision Pro! – Virtual Reality Oasis

On YouTube, you can find filmmaker Hugh Hou's free VR videos for everyone to enjoy Experience 8K 3D VR180 – Hugh Hou, and his channel also has other 360VR videos.

These YouTube 360 videos, when viewed in Safari, look like the image below. You can move the viewpoint, but you cannot switch to full screen or immersive mode. The author suggests that Apple Vision Pro users download their app to watch the videos for the best viewing experience.

WebXR Requires Enabling Experimental Features

Headsets have WebXR and WebVR APIs, but Safari's support is particularly limited.
If you want to play with WebXR content on Apple Vision Pro, you must go to Settings > Apps > Safari > Advanced > Feature Flags, and then enable WebXR Device API and WebXR Hand Input Module. This forces the use of experimental features that the system has not officially released.

Then you can visit Immersive Web Developer Home or other sites to explore their demo projects. Before proceeding, you will see system warnings about limitations and operational instructions.

Then, when you operate it, the entire screen is divided into a designed 3D scene.
If the angle is incorrect, moving forward might lead to a black area, which could be the boundary or the edge of the scene.

Parallax Effect

This is not a sci-fi channel; all objects in the scene stay obediently within the small frame of the Apple Vision Pro Safari browser. Trees, houses, or any other objects in the scene will not extend beyond the browser frame. The background of the scene is also white, so you won't see through to the outside.

Common parallax effects generally fall into two categories:
1. Parallax Scrolling:
The background image is divided into multiple layers, and as the screen scrolls, each layer moves at a different speed (as shown below).

Apple Vision Pro allows you to scroll the screen using Pinch and drag (pinching and dragging up and down with your fingers) or swipe (quickly swiping),
compared to using a mouse wheel where your fingers barely move, or swiping on a small mobile touch screen,
scrolling on Apple Vision Pro feels a bit different,
it requires more muscle movement and is more tiring to scroll the screen.

2. Mouse move parallax:
Similarly, objects are divided into multiple layers, and when the mouse cursor moves, the cursor position is obtained, and then the movement position of each layer is specified (as shown below).

And because Apple Vision Pro is a system closer to touch devices, not only does it lack a cursor, but it also seems that even if you glance around or nod, it cannot trigger mousemove events to continuously get the cursor position, so these effects are also rendered useless.

However, when it comes to developing parallax effects, besides capturing scroll events and getting the mouse cursor position during movement, there is a third method that uses the information from the device's accelerometer. But mobile Safari's support for the HTML5 Accelerometer is also quite limited, and even the browser's request for permission window doesn't appear, so don't expect too much.

visionOS Safari's UserAgent String

visionOS 1.1 Safari's UserAgent string is as follows:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15

Although the device was set to visionOS 1.1, the UA string does not contain any mention of visionOS.
At the time of writing, the official iOS version is only up to 17.3, and 17.4 is in beta.

If you click the AA button in the address bar,

select Request Mobile Website

or go to website settings and turn off Request Desktop Website (the default setting is on)

The UA string will change to:

Mozilla/5.0 (iPad; CPU OS 16_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Mobile/15E148 Safari/604.1

This screen is from a test done back in 2019/8 when iPad started supporting mouse, and it's still relevant now...

Safari Windows Cannot Be Maximized Indefinitely

The first time you open Safari, or every time you open new windows,
the Safari window is not in a 16:9 ratio or anything like that, but rather 1024*890 px.

The window can, of course, be resized by dragging the control point at the bottom right corner, which feels similar to the screen angle on iPad and macOS.
The maximum size can be dragged to about 1782px*1188px (3:2),
and the minimum can only be reduced to about 829px*743px.

Some websites insist on having a specific layout or arrangement for the first screen, and under these specific ratios, it might display very strangely.

visionOS Safari, like other Apple devices, does not show scroll bars while scrolling; the scroll bars will hide, unlike Windows where a gray bar always hangs on the right side.
Users can, of course, “zoom in” to enlarge the screen (zooming in and enlarging by pressing +- % are displayed differently).
Combining the characteristics of scroll bars, the screen might appear cut off,
and basic web designers might have to spend hours communicating, asking clients to zoom out themselves, or pretending to fix something but actually doing nothing.

Some websites might ignore mobile-first index SEO strategies, or have separate domains for mobile and desktop versions.
Apple Vision Pro's Safari window width usually won't trigger the mobile version,
and because the UA string is set the same as macOS Safari,
users might be forced to operate the desktop version of the website on a large screen using touch controls.

Android and other platforms have supported the HTML5 Fullscreen API for many years, and it has also started to be supported on iPad.
So, visionOS Safari will likely follow iPad in supporting it,
or will it follow iPhone in not supporting it? It's worth looking forward to.

Browser Interaction Events Supported by Safari

For precision control, from the motion control setups seen in childhood like Wii and PS Move VR, to some recent VR headsets, product packages usually include some oddly shaped control handles. However, Apple Vision Pro did not start with these; it was designed to be operated using hand movements and eye tracking.

In some unboxing videos, it can be confirmed that Apple Vision Pro can also use Bluetooth to connect to Magic Trackpad and wireless keyboards, but let's assume users are using hand gestures for operation.

In this test, unsurprisingly, mouse hover and mouseover CSS effects do not work,
if a website has a corner where hovering the mouse would slide out a navigation path or logo list; on a small mobile screen, it would be designed to expand with a tap, and then collapse with a close button.

Users browsing on Apple Vision Pro will likely first stare at the object, then pinch with their fingers to make that element slide out; due to the window width, the large screen version web page does not have a close button, so they can only stare at the object and pinch again, hoping that the list will retract...
If it doesn't work, they might have to stare at some empty space and pinch to make the object lose its hover state, so that the list can retract.

Forced Highlight Styles

Another issue is that elements on the website that are being stared at will be forcibly given a light semi-transparent style by the system:

As shown above, that white semi-transparent box is not something I set. The website already has some hover, focus, active styles set, but Apple forcibly adds another layer of white semi-transparent box, which sometimes looks a bit off.
Using -webkit-tap-highlight-color also doesn't cover it. If I find a way to modify it later, I'll update.

Additionally, currently, visionOS Safari seems unable to directly inspect websites,
I don't know how to bring up Safari's Developer Tools,
browser settings not enabled? Can't find that option in system settings>Apps>Safari.
Previously, in macOS Safari, you could connect to the website inspector opened from the Xcode device simulator, but this time, the visionOS device didn't appear.

visionOS Safari Browser Features

Currently, possibly because the hardware is only in its first generation and visionOS is only at version 1.1, the entire Safari functionality is somewhat simplified,
but only some features are simplified, while privacy features are not lacking (as shown below)

The browser's most important web display functionality is also top-notch

CSS3 rendering effects are perfectly passed (as shown above), easily surpassing IE.

Chinese webfonts can also be displayed normally (as shown above)

Conveniently tested with HTML5test, scoring over 500 points, which is quite close to mainstream browser levels.
Which items didn't score full marks? See below for yourself.

visionOS Safari's Form UI

For visionOS, Apple HIG has added a set of Designing for visionOS and a component library, and the UI has this semi-transparent texture.

As mentioned earlier, users might see websites designed for desktop or large iPad window widths,
but they can directly walk up to interact with it,
or use eye tracking to focus on it, then pinch with their fingers,
so some native web elements also appear more like those for touch devices.

Here are some observed examples:

As shown above, when typing in a form, the keyboard sometimes floats directly below the input field,
Additionally, the textarea originally had a control point in the bottom right corner to adjust the size of the input field, but in visionOS Safari, it is mysteriously missing?

As shown above, the spacing of the dropdown menu (select) is set very large

As shown above, other input types in visionOS system UI look like this.
Checkbox, radio, range, and those more common ones are not shown here.

Some popular apps do not support Vision Pro

This article uses the Xcode device simulator to run visionOS, which is for developers to test their own apps. There is no App Store, so there is no way to get other people's apps or native apps. There is no special way to get a set of instructions.

Although there are more than 600 new apps built specifically for Apple Vision Pro, and there are more than 1 million compatible apps in the Apple announces more than 600 new apps built for Apple Vision Pro, which are compatible with iOS and iPadOS App Store, but according to MacStories review, among the 40 most popular movie/social/news/sport/game/shopping/working apps on the App Store, there are few native apps that support VisionOS, there are some app don't allow to install.

Apps like Facebook and Instagram are not available in Apple Vision Pro, so you must use Safari to access those apps you used to use on your iPad/iPhone.

YouTube:
YouTube says a Vision Pro app is on the roadmap
As of January 2024, it's not supported, but as of February 2024, it's on the roadmap.

Netflix:
Our members will be able to enjoy Netflix on the web browser on the Vision Pro
It's not supported, according to Kumiko Hidaka in an interview, but it's expected that Netflix can be used in the Vision Pro web browser.

Spotify:
Closed Ideas – Spotify Community – Basic support for vision pro
It's not supported, and the user community has closed the request, indicating that there is no plan to support it in the near future.

Disney Plus:
Apple TV+ and Disney Plus are also on the list of Apple previews new entertainment experiences launching with Apple Vision Pro, and Apple has announced that there are more than a dozen apps on the list, including Amazon Prime Video, Paramount+, Peacock, Discovery+, etc., but Netflix is excluded. It seems that Netflix not releasing an app is due to some unresolved negotiations between the two companies, leading to a standoff.

Microsoft Office:
Announcing Microsoft 365 apps available on Apple Vision Pro
Is Vision Pro just for show? The most powerful productivity suite, Microsoft, announced in February 2024 that Microsoft Teams, Word, Excel, PowerPoint, Outlook, OneNote, and Loop will all be available on the Apple Vision Pro App Store.

Limitations

After being baptized by touch devices on mobile phones and tablets in recent years, the general principles of web page or UI design are the same as before.
– Stop asking designers to create a bunch of mouse-hover effects.
– When making tooltips, don’t use ones that can only be seen by hovering the mouse cursor.
– Not only the browsing and viewing functions, but also if there are functions for people to touch, don’t make the UI too crowded.
– Don’t just design for 1920 and 375 widths and think that it can fit all widths of windows.

Unless the content of the web page is very attractive and valuable, users have to take off their helmets and take out their mobile phones or computers to read web pages? It sounds very inconvenient.

These topics of AR, MR, XR have been the subject of many people's university graduation projects for decades. From the ingenious ideas at the new generation design exhibitions to the recent metaverse or Figmin XR, everyone is talking about the potential future that can be expected. But when will this future come?
Are we waiting for the hardware to be good enough to completely solve problems like being too heavy, poor image quality, and high power consumption?
Are we waiting for 3D and programming development to become basic skills, just like junior office staff can also edit images, cut videos, and make web pages?
Are we waiting for the ecosystem to be perfected? Developers' time is spent on the product content itself, not dealing with the compatibility of N platforms and systems; or applying for various accounts to charge users 1 to 3 dollars.

Apple Vision Pro and visionOS may make waves in this field again. As for how long they can last, they will always be toys for a small number of people. Let’s continue to see.

Another interesting question
For apps like LINE, one account can only be logged in to one computer + one iPad + one mobile phone at the same time.

What should I do if I encounter this thing?
(A) Install the LINE app in visionOS, and then LINE on the iPad will be logged out (but currently it is not known whether the Deep Link behavior of visionOS is the same as that of iOS. Maybe even if the LINE app is installed in visionOS, the link to add friends will still appear) Cannot automatically jump to LINE APP).
(B) Follow Cheri’s example and put your phone into the helmet to scan the QRCode.
(C) Use AirPlay to project the Apple Vision Pro screen to a TV or surrounding devices, and then scan it with your phone.
(D) Please modify LINE’s cross-device login and friend-adding mechanisms,

Further reading

• Apple Vision Pro User Guide- Safari Official operating instructions , there are several chapters in the selected table of contents.
• Meet Safari for spatial computing A Session at WWDC 2023, a 16-minute video, There are some introductions to the development of the visionOS Safari browser.
• Something You must know before landing on visionOS Yes A web front-end senior watched the above video and sorted out some front-end development matters that may be different.
• Designing for visionOS Maybe 99% of designers don’t have chance to design a visionOS app. If you meet a team that wants to develop a visionOS app, no one will listen to the UI designer if they really want to develop it, but you can still take a look.
• Apple Design Resources – visionOS Component library in Figma file format
• visionOS Developer Documentation Why would someone who isn’t an app engineer need to read these development documents? Sometimes, people who instruct engineers to do things have some ideas that are irrelevant and unrealistic. One of the sources of these problems is not reading the documents…...
• A Survey of Popular Apps Currently Compatible With Apple Vision Pro BY JOHN VOORHEES There is a table listing which apps cannot be used with Apple Vision Pro.
• UX Case Study: AI-Powered Spatial BankingUXDA made a virtual design case for the Apple Vision Pro banking app, with a lot of design thinking and future prospects.