Will Apple's new technology open the door to a futuristic experience?
0 Facebook x.com Reddit
Apple has filed patent applications for many technologies that at first glance do not seem so interesting. Together, they paint a picture of a world where Apple Vision Pro and augmented reality capabilities can be accessed anywhere.
In Free Guy, Ryan Reynolds plays Guy, a non-player character (NPC) in the world of video games. He is destined to live the same day over and over again as an unremarkable teller in an unremarkable bank that is repeatedly robbed by criminals in dark glasses.
In the first scene of the film, Guy explains that sunglasses wearers can do whatever they want and are seen as all-powerful heroes. Unbeknownst to him, these influential people are avatars of real players of the console game he lives inside.
During one of the daily bank robberies, instead of dutifully following the sunglasses-wearing robber's orders, Guy fights back and takes his sunglasses. After putting on his sunglasses when leaving the bank, Guy is suddenly able to see the world around him in the same way that the players do.
Can glasses (or a headset) change your world? Courtesy of 20th Century Studios
This is a Fortnite-style collage showing the player's health bars. A floating medkit restores Guy's health, and clothing and weapon upgrades are sold everywhere he looks.
Reading patents doesn't usually remind me of movies. However, in a recent roundup of Apple's weekly patent updates, I found several related applications that collectively hint at Apple's plans to bring the film to life— at least to some extent.
Apple patents for changing AR
Two interesting patents: “A system for determining position both indoors and outdoors.” outdoors, The second is “Notification of augmented reality content on an electronic device.”
The most important thing is the “inside-outside position detection system.” This patent describes a system for pinpointing a user's location using a wide range of devices in addition to Apple's Ultra Wideband chip. In practice, devices such as Wi-Fi routers can already determine a person's location, but this is usually used for general location determination.
Triangulation allows you to get a general idea of a person's location. Apple's patent adds some features that make location determination much more accurate by using various additional types of wireless devices.
The more signals available, the more accurate the training will be, just as GPS becomes more accurate the more satellites it detects.
The patent describes typical wireless devices that are measured and evaluated simultaneously to get a more accurate picture of someone's location in a given space. Wi-Fi and Bluetooth transmitting devices can already be used in tandem, but this patent adds something interesting.
This illustration from Apple's patent application shows several cameras used to determine location.
Apple plans to use cameras that track users and transmit location data . In practice, this technology is similar to that used in the Amazon Go store. In Amazon's case, multiple cameras track shoppers from different angles to operate an automated checkout system that charges users for what they see on the store's computers.
Instead, using this Apple system will locate the user in a physical space to display mixed reality [MR] content. Currently, Apple Vision Pro uses its pass-through cameras and physical sensors to determine where the user is, including where they have placed things like displays or “hanging” artwork.
Apple's technology will allow MR objects to be placed in any environment without relying on image recognition to track objects.
The use of multiple small devices can provide a “real world” interactive experience with ultra-precise user positioning. As each device was added to the review, the quality of service improved. I can imagine some pretty interesting uses for this technology.
Just like Free Guy, immersive interactive experiences will be possible with Apple Vision Pro and future devices. Unlike Free Guy, this technology can improve the user's interaction with the world, which is more like a William Gibson novel — hopefully without dystopia.
Indoor accuracy for better AR experiences
There are obvious use cases in retail, although this is the most common use case. For example, you can walk into an Apple store and see the iMac's specs hovering above it, or pick up an iPhone case and overlay images of it on the phone to see what it looks like.
The ability to overlay MR content onto the physical world is not new and is already being used in everything from gaming to operations.
Team of surgeons with (left) Apple Vision Pro (source: London Independent Hospital, Daily Mail)
Home team advantage h2>
The idea of an exciting sporting event is more interesting. The main advantage of watching the game on TV rather than on the court is that the broadcasts include real-time statistics, information about the players and the game, and commentary from experts.
Overlaying statistics while watching a game is nothing new, but this position-aware technology can provide ultra-precise positioning of MR content. Instead of generalized MR elements, being able to pinpoint the observer would open up some interesting possibilities.
The main thing about this patent is to extend this idea to the environment. Typical outdoor spaces lack multiple Wi-Fi points, limiting triangulation capabilities. Low-power Bluetooth devices can be added to this mix to radically improve location detection.
Enabling cameras that transmit angular view data will allow someone to set up multiple devices and create a location-based environment. Apple iSpy may be coming soon.
One of the best magnetic resonance imaging technologies used in sports today is the virtual first down line used in broadcasting football games. The down marker is easy to see in person, but it's hard to visualize on TV where the offensive team needs to get to in order to get a first down.
Although you can overlay data on the real world using Apple Vision Pro and Meta Quest 3, this information cannot be tied to a specific location. In other words, you can overlay general statistics about the game, but you can't target them to the user's location.
In this example, the first down marker can be visible no matter where someone is sitting. Whether it's behind the end zone or in the stands at the 50-yard line, the baseline can be accurate from the user's perspective.
Real-time data could easily be overlaid on individual players, following them around the field no matter where someone is sitting. For example, if you want to know the statistic of a baseball player's batting average, it can be placed above the player no matter where the visitor is sitting.
The ability of a venue to add more transmission points, such as additional Wi-Fi devices and cameras, will allow a venue or retail store to add simple devices and provide an immersive experience.
These examples show AR content notifications for TV shows or movies, but can be expanded to cover more interesting AR use cases.
Find AR near you
This is related to the second interesting patent “Notification of Augmented Reality on an Electronic Device. “This in itself looks like a way to trigger an alert on your phone when an MRI machine is nearby.
As the name suggests, the intersection of this and the first patent will allow someone to know if they are in an immersive environment. Notifications will presumably appear on the Apple Watch or iPhone, and most people will currently use them on the iPhone since it can already create mixed reality.
These patents are not the first to point to a more immersive world of augmented reality, although they are the first to describe expanding the experience globally.
Apple is looking to improve the accuracy of MR headsets, and an interesting new patent application points to other ways to track a user's position outdoors or indoors using the right condition.
This patent uses “celestial objects” to help navigate a 3D environment. Most heading systems include an electronic compass and recognition of observable objects such as furniture and walls. This patent will include the position of the Sun and Moon relative to the user to ensure high accuracy.
The patent shows the sun being detected through a window in a room, although it also describes its use outdoors. In the stadium MR example, the position of the sun overhead provides strong contextual clues about where the user is looking.
Of course, this will also require the device to have access to ultra-precise information about the position of the sun relative not only to the time of year, but also to the user's altitude.
By using the position of the Sun or Moon, Apple plans to improve positioning accuracy for AR users
To improve the user experience in AR and provide more efficient ways to interact with the world and virtual elements, Apple has filed a patent describing additional gestures in addition to the pinch gesture.
Unlike other Apple patents, this one describes ways in which multiple AR users can interact with both physical and generated 3D objects. For example, this technology will allow multiple Apple Vision Pro users, and even iPad or iPhone users, to manipulate virtual objects in the environment, each looking at the object from a different perspective and each appearing correctly oriented.
Wearing Controllers
Apple will likely combine these technologies with several additional patents, including “Cable Actuated Items”, “Fabric Controller” and the rumored Apple Ring.
The Apple Ring will almost certainly interface with the Apple Vision Pro, possibly providing even more precise tracking and an additional control surface.
Apple has been working on smart rings since at least filing a patent in 2019.
The first patent we looked at when it was filed in 2020, would allow the integration of gloves or other devices with a sensor. This patent may have been early research during the development of Apple's Vision Pro, but it shows that even four years ago Apple was on its way to unconventional ways to control devices.
The second patent describes the technology of using fabric to cover electronic buttons. An obvious application for this patent would be a couch potato-like device with buttons on a fabric-covered surface. This could be combined with the glove patent to provide a more tactile user experience and possibly haptic feedback when using Apple AR/MR tools.
We are just at the beginning of the exciting world of AR/MR, currently led by the Apple Vision Pro. While we're a long way from widely adopting XR capabilities and portable hardware to expand their capabilities, we see the path Apple is taking to make XR a part of our everyday lives.
