Share content found on your screen in apps with Apple Intelligence with upcoming APIs
0 Facebook x.com Reddit
Siri will soon be able to view and act on your screen with new developer APIs built on technologies we’ve talked about in AppleInsider reported ahead of WWDC.
Apple released new documentation on Monday to help developers prepare for upcoming Siri and Apple Intelligence features. The company's latest developer API reveals that Siri will gain significant contextual awareness and that the virtual assistant will at some point be able to use information from the content currently on the screen.
Siri will undoubtedly become much more useful thanks to Apple’s changes. The company has provided a list of examples that give some idea of what exactly the new and improved AI-powered Siri will be able to do in the future.
Users will be able to ask Siri questions about the webpage they are currently viewing or a specific object in a photo. The virtual assistant will also be able to summarize documents and emails on request or complete texts by adding more content.
Note that some of these features were already implemented in the first iOS 18.2 developer beta, which introduced ChatGPT integration. Siri can forward a PDF, text document, or image to ChatGPT for certain actions, but only with the user’s permission.
The new developer API suggests that Apple wants to simplify this process even more. Instead of asking Siri to send a document to ChatGPT, the user will be able to ask direct questions about the page on the screen or use information from it in some way. There is a lot of room for improvement here, as ChatGPT can currently only access screenshots or documents manually provided by the user.
Siri may soon be able to use on-screen content.
Apple’s idea of having AI use on-screen information was clear even before Apple Intelligence was announced at WWDC. Research published by the company, especially around the Ferret model, served as an indicator of Apple’s plans for artificial intelligence.
There was a significant focus on document analysis, document understanding, and text generation using AI. In one of our recent reports, AppleInsider described various internal test apps that were used while Apple Intelligence was still in development.
The test apps and environments, particularly the 1UP app, showcase many of the features that are now possible thanks to the ChatGPT integration in the iOS 18.2 beta. Apple also had a dedicated app for testing smart replies in Mail and Messages.
Siri’s new ability to complete and summarize text or answer questions about images, documents, and web pages was also teased before the official announcement. In our reports on the Ajax LLM, as well as the BlackPearl and Greymatter projects, we introduced many of these features, explained how they would work, and even paraphrased Apple’s AI hints.
It’s clear that the iPhone maker is taking AI quite seriously, given the amount of time, research, and effort it’s putting into its generative AI projects. The developer API was released on Monday only to help developers prepare for new Siri features rumored to debut in 2025 with the iOS 18.4 update.
Follow AppleInsider on Google News
