This issue's cover is from Nikita Belousov's design of the visionOS ARC browser concept, where the essence of visionOS in a three-dimensional space is fully considered, giving the Arc browser's Space and Pined Site a sense of depth.
Table of Contents
BigNews
visionOS 1.1 and Xcode 15.3 released!
PICO updated to 5.9 system, bringing PICO Connect and Avatar SDK
Weekly App
MagicRoom: Add some magic to your room
Share your space videos on Spatial Station, ImmersiShare, and Share Spatial!
Tool
Quickly search for visionOS apps on Mac
Video
GazeTrak: Eye tracking with sound?
Article
The first 1000 apps on visionOS
Apple Vision Pro has great potential in the healthcare field
Create With Swift-Make it Spatial: High-quality Swift and visionOS development tutorial articles
SmallNews
Perhaps, the lifelike environment in Apple Vision Pro is not just panoramic video
Quick News x 9️⃣
BigNews
visionOS 1.1 and Xcode 15.3 Released!
Keywords: visionOS, Xcode
After over a month of beta testing, visionOS finally released the official version 1.1 this week. Let's highlight some interesting points of this update~
First, a very useful update not mentioned in any update notes: visionOS 1.1 solved the issue of screen recording in low light conditions. Previously, if your room was dark, you couldn't screen record (unless you turned on travel mode to bypass this), as complained by Reddit users.
Now in visionOS 1.1, we can still record the screen even in almost pure black rooms.
In addition to solving the screen recording problem, in the new version of the system, if you want to record Persona, you can choose not to hold the Apple Vision Pro yourself.
Instead, you can choose the Hand-Free method in Accessibility, place the Apple Vision Pro somewhere, or have someone else hold it for recording.
One might ask, what's the point? Likely, this reflects Apple's consistent positioning of its products: making them accessible to a diverse range of people. As shown in the Equal Accessibility video, even a user without hands can effectively use the Apple Vision Pro. Knowing this, it makes sense why Apple Vision Pro specifically provides a Hands-Free Capture option.
Next is the improvement of Persona itself, which is subjective. Have a look at the comparison by NewAppleTimes, do you think Persona has improved?
The official update notes indicate that this update mainly targets the appearance of hair, makeup, neck, and mouth. By the way, this update also brought improvements to the look and stability of EyeSight, Mac Virtual Display (the function of projecting Mac to Apple Vision Pro), and the reliability of the virtual keyboard.
Moreover, after this update, all official apps except for Settings, Photos, Safari, App Store, Messages can now be deleted. (This logic is more like the iOS logic that wasn't implemented before.)
Additionally, following the visionOS update, Xcode 15.3 was also released, which didn't involve much content related to visionOS development. Perhaps the only relevant feature is the hidden "Export Preview Screenshot" in Editor > Canvas, allowing direct export of app screenshots from Xcode's Preview.
Overall, visionOS 1.1 and Xcode 15.3, as a minor version update, didn't bring very rich content, more like some basic feature optimizations.
PICO Updated to 5.9 System, Bringing PICO Connect and Avatar SDK
Keywords: PICO, Unity, Avatar SDK
After nearly half a year, PICO finally welcomed a new system upgrade – 5.9.0 (the last upgrade to 5.8.0 was in October '23). On the user side, PICO brought a major feature: PICO Connect. This is an upgraded product of the original streaming assistant.
With PICO Connect, Windows and Mac users can mirror computer screen content to PICO (similar to Mac Virtual Display on Apple Vision Pro), and on Mac, PICO Connect also supports switching between multiple connected screens:
Currently, PICO Connect seems usable in terms of stability and clarity (supporting up to 4K resolution (3840*2160), screen ratio of 16:9), but now due to the lack of a mature multitasking mechanism in the PICO system, other apps can't be opened for simultaneous operation when PICO Connect is active. Hopefully, this will improve in future versions.
However, thanks to the "Auto Connect to This Computer" setting in PICO Connect, your computer will automatically project to PICO when you put it on, which feels very convenient.
In addition, PICO has updated video playback and gesture recognition to some extent.
In terms of video playback, PICO mainly improved the recognition accuracy of its built-in video player. When you open various 3D, 180°, 360° videos in PICO's file manager, it will automatically help you identify the video format and play it correctly. Also, the new "Premium Image Mode" enhances the sharpness of 4K videos during playback.
For gesture recognition, PICO mainly optimized for some edge cases:
On the developer side, the newly released PICO Unity Integration SDK 2.5.0 now supports the XRI framework and Hands package.
Editor's Note
As early as June '23 in 002-Unity XR Interaction Toolkit released version 2.3, we introduced XRI and Hands, saying that this feature requires platform support. Nearly a year later, PICO finally supported it (official documentation here), feels like a wish come true 🫣
The PDC tool also now supports hand gesture preview:
On another note, the newly launched Avatar SDK fills a gap for developers on PICO's virtual human aspect. Previously, developers on the PICO platform could only create or use third-party SDKs like Ready Player Me to create avatars for users. PICO previously provided users with an avatar editing feature (click "My" -> "Edit Virtual Image" to enter this interface):
However, developers previously had no access to these avatars. This Avatar SDK now provides these avatars (including full-body skeletal models) to developers. With the Avatar SDK, developers can directly access these avatars like other PICO platform services (such as accounts), and can pop up similar UIs in the app for players to add or edit avatars:
If you want to know how to use the Avatar SDK, you can check out the sample project provided by the official on Github.
Idea
MagicRoom: Add Some Magic to Your Room
Keywords: The Matrix, visionOS
MagicRoom is a visionOS app that adds a layer of "spatial filter" to the real world. Using the Scene Reconstruction capability provided by visionOS, MagicRoom can make your surroundings look like they are covered with a colored grid:
Or like the digital rain in The Matrix:
Although currently there are only these two "filters", it is believed that more similar filters will be added to MagicRoom in the future (finally, our imagination is the only limit to the final effect😆)
Share Your Space Videos on Spatial Station, ImmersiShare, and Share Spatial!
Keywords: Space video, Sharing
Thanks to the user In the Clouds for the recommendation
Watching space videos on Apple Vision Pro truly feels like going back to those particular moments in the past. Now, by watching space videos shared by friends around the world, we can experience interesting activities and beautiful scenery from all over the world without leaving home. If you have captured wonderful moments in life, consider sharing your space videos! Currently, there are 3 ways to share space videos:
Spatial Station is an already launched app with a fairly complete user system and many user-uploaded space videos. The homepage has sections like Latest Uploads, Current Hot, Hot Trends, etc. However, the main issue is the varying quality of user-uploaded videos (such as some with severe shaking causing viewer dizziness, and some with less prominent parallax and lacking a strong 3D feel).
Another app, ImmersiShare, also allows for the sharing of space videos. This app is special in that the videos shared are actually uploaded to the developer's iCloud for sharing, so the loading speed of the videos is subject to network influence.
Apart from app-based sharing, Share Spatial is a website where you can find space videos, space photos, and even panoramic images. There are even some artists sharing their panoramic images, which are quite stunning. Share Spatial is also testing their own app, and soon everyone will be able to share and view more conveniently on Apple Vision Pro.
Tool
Quickly Search for visionOS Apps on Mac
Keywords: Raycast, vision.directory
The vision.directory we recommended in Issue 019 recently added a plugin on Raycast, a highly praised Mac launcher, allowing you to search for visionOS apps directly through Raycast.
Video
GazeTrak: Eye Tracking with Sound?
Keywords: Sonar technology, Eye tracking
Eye tracking is a key technology in XR, used for focus rendering in the user's gaze area to reduce system rendering pressure, and also for UX design, like the overall system interaction in Apple Vision Pro.
Although relying entirely on eye tracking for system interaction can cause problems, such as rapid fatigue for interactions requiring frequent, fast switching, and a difficult balance between providing APIs for developers to unlock more possibilities and user privacy.
However, achieving precise eye tracking using cameras is a significant challenge. Recently, researchers from Cornell University in New York publicized a technology that uses sonar-like technology instead of cameras for eye tracking: GazeTrak. This technology utilizes 8 miniature speakers, each capable of emitting sound at frequencies above 18 kHz.
As these signals are beyond most people's hearing range, they are imperceptible to humans and also immune to noise interference.
The research team suggests that sonar technology should have several advantages: compared to camera-based systems, GazeTrak can operate at low power and, since it doesn't require continuous data recording, offers more privacy for users. Additionally, it could reduce the manufacturing cost and weight of VR headsets. However, a main drawback of GazeTrak is that enough data needs to be collected for model training due to the different eye shapes of individuals.
Article
The First 1000 Apps on visionOS
Keywords: visionOS, AppStore
In The First 1000 Apps on visionOS, the author analyzes the 1217 apps currently on the AppStore, offering simple insights into their types and pricing.
It's evident that compared to other XR devices, apps on visionOS are still mostly productivity and tools focused, highlighting the difference between Apple Vision Pro and other devices—it's not a gaming console, but a spatial computing device.
Apple Vision Pro Has Great Potential in the Healthcare Field
Keywords: Apple Vision Pro, Healthcare
Utilizing XR to assist doctors and nurses in surgery has always been an exciting concept, but due to various reasons, XR technology has never been fully applied in real surgical environments, until recently. According to a report by Daily Mail Online, in a real surgery in London, nurses working with surgeons used Apple Vision Pro to help prepare, track the surgery, and assist in selecting the right surgical tools.
For instance, myMako allows surgeons to easily share the details and insights of complex knee surgery plans in a 3D, intuitive, and dynamic way.
CyranoHealth simulates the use of medical equipment, helping frontline workers like nurses familiarize themselves with new and complex medical devices (such as medical infusion pumps), thereby reducing work-related anxiety.
Epic Systems allows doctors to perform tasks like gesture drawing, patient communication, and condition analysis in Apple Vision Pro, opening new collaborative and productivity workflows for future medical applications.
Apart from assisting doctors in their tasks, Vision Pro can also help us pay attention to our physical and mental health. For example, Cedars-Sinai's Xaia app provides intelligent conversational mental health support to users, who can also engage in deep breathing exercises and meditation.
There are many more healthcare apps, such as Visage Ease VP for improving patient care and aiding in medical data review for inefficient tumor committees; Falcon Vue aims to empower doctors with the ability to view all medical imaging data; Insight Heart allows us to understand our heart in 3D and explore various heart conditions. Cinematic Reality lets surgeons, medical students, and patients view medical images obtained through medical scans in the real environment.
Nature Magazine also notes the great potential of Apple Vision Pro in the healthcare field. For example, the advanced eye-tracking technology of Apple Vision Pro might detect early signs of motion sickness, stroke, or dementia. Surgeons can enhance their awareness of patient information by wearing Apple Vision Pro, such as seeing the patient's vascular structure during surgery for more precise operations.
Create With Swift-Make it Spatial: High-Quality Swift and visionOS Development Tutorial Articles
Keywords: Swift, visionOS, Spatial, Accessibility
Create With Swift is a website maintained by developers and designers at imaginary.institute, sharing many development and design insights on Swift app development. The website is mainly divided into two sections:
Make it Accessible: Mainly sharing design and development articles related to accessibility features
Make it Spatial: Mainly sharing design and development articles related to visionOS
Thanks to the collaboration between design and development, the content on the website is concise and useful, often accompanied by attractive illustrations. For example, this article discussing the design of icons on visionOS uses an animated GIF to represent the feel of Spatial Icons, which is both interesting and visually appealing:
Small News
Realistic Environments in Apple Vision Pro Are Not Just Panoramic Videos
Keywords: Apple Vision Pro, Environment
Environment is one of the few official Fully Immersive experiences on Apple Vision Pro. Everyone who tries Apple Vision Pro for the first time is amazed by the detailed quality of the scenes and the surrounding spatial audio in the environment. Many think these are panoramic images or videos, but these might actually be meticulously modeled scenes by Apple! Don't believe it? Check this out:
Through M1's roaming perspective, we can see that the entire scene is indeed modeled, and to reduce unnecessary computational load, modeling and texturing are omitted in areas not visible to the viewer. Once again, we are in awe of the immersive experiences Apple Vision Pro offers!
XR 基地
