If you haven't watched yesterday's OpenAI event, I highly recommend doing so. The big news was that the latest GPT-4o model works seamlessly with any combination of text, audio and video.
This includes the ability to ‘show’ The GPT-4o app is a way to record another app's screen as you film it – and that's exactly what the company demonstrated with an incredible AI learning demo for the iPad …
GPT-4o
OpenAI reported that ‘o’ means ‘omni.’
GPT-4o (“o” instead of “omni”) is a step toward a much more natural human-computer interaction: it takes as input any combination of text, audio, and image, and generates any combination of text, audio, and image output.
It can respond to audio input in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to the reaction time of a human (opens in a new window) in a conversation […] GPT-4o is particularly better at sensing image and sound compared with existing models.
Even the voice aspect is of great importance. Previously, ChatGPT could accept voice input, but would convert it to text before working with it. GPT-4o, on the other hand, actually understands speech, so it skips the translation step entirely.
As we noted yesterday, free users also get a lot of features that were previously only available to paid subscribers.
AI Demonstration for iPad
One of the capabilities demonstrated by OpenAI was GPT-4o's ability to look that you do it on the iPad screen (in split-screen mode).
The example shows how AI teaches a student who has solved a math problem. It can be heard that initially GPT-4o understood the problem and wanted to solve it immediately. But the new model can be interrupted, and in this case she was asked to help the student solve it himself.
This demo is insane.
A student shares their iPad screen with the new ChatGPT + GPT-4o and the AI talks to them and helps them learn in *real time*.
Imagine giving this to every student in the world.
The future is so bright. pic.twitter.com/t14M4fDjwV
— McKay Wrigley (@mckaywrigley) May 13, 2024
Another possibility that can be seen here is that the model claims to detect emotions in speech and can also express emotions itself. This was a bit overdone in the demo for my taste, and that's reflected here – the AI is perhaps a bit forgiving. But this is all customizable.
Basically, every student in the world can have a private tutor with such capabilities.
How much of this will Apple include?
We know that artificial intelligence is a major focus of iOS 18, and that it is finalizing a deal to bring OpenAI features to Apple devices . While this was described at the time as a proposal for ChatGPT, it is now likely that it is actually about access to GPT-4o.
But we also know that Apple is working on its own artificial intelligence models . , with its own data centers using its own chips. For example, Apple is working on its own way to allow Siri to understand app screens.
So we don't know exactly what GPT-4o capabilities the company will implement into its devices. but this one seems so Apple perfect that I have to believe it will be included. It's really about using technology to empower people.
Image: OpenAI. Benjamin Mayo contributed to this report.
