By Julie Clover
Apple is working on more advanced Siri functionality as part of its Apple Intelligence feature set, and is providing the App Intent API to developers to prepare for the new capabilities.
With the latest wave of betas, Apple has a new API that allows developers to make on-screen content in their apps available to Siri and Apple Intelligence. From Apple's documentation:
When a user asks a question about on-screen content or wants to perform an action on it, Siri and Apple Intelligence can retrieve the content to answer the question and perform the action. If the user explicitly requests it, Siri and Apple Intelligence can send content to supported third-party services. For example, someone could browse a website and use Siri to provide a summary by saying or typing something like, “Hey Siri, what’s this document about?”
In iOS 18.2 beta, ChatGPT’s integration with Siri allows users to ask questions about photos and documents like PDFs and presentations and get information about them. For example, you could ask Siri “What’s in this photo?” and Siri will take a screenshot and send it to ChatGPT. ChatGPT then relays what's in the image, and the same feature works for PDFs and other documents.
It doesn't appear that iOS 18.2's ChatGPT integration is the screen awareness feature Apple has planned for Siri, but it could be related. Apple describes screen awareness as Siri's ability to understand and take action on things on the screen. For example, if someone sends you an address, you'll be able to say “Add this address to their contact card,” and Siri will do so. The feature isn't available in iOS 18.2, and ChatGPT is limited to evaluating screenshots, but that's a bit confusing.
Screen awareness, like personal context and in-app actions, is a feature Apple has planned for Siri, but we probably won't get it this year. Many of Siri's features are coming in a future version of iOS 18, and Bloomberg's Mark Gurman said we can expect them in iOS 18.4, an update due in spring 2025.
While there are a few Siri features that aren't available, we'll be able to see them in the future. For features that won't be coming until next year, Apple is providing APIs to developers early so that developers have a few months to prepare, and so that the features are ready for the public when the updates actually come out.
Related reviews: iOS 18, iPadOS 18Related forums: iOS 18, iPadOS 18[ 26 comments ]
