Loona & Moxie to be turbocharged with OpenAI’s new ChatGPT-4o “Omni”

[Chris]

This will blow your mind. “Her” is literally a reality.

OpenAI just released this presentation featuring two smartphones operating OpenAI’s new ChatGPT-4o.

The two AI’s are shown talking to each other, singing and interacting with the world through audio, vision and text in real time using a single AI model to perform all processing.

The speech used by the AI is so natural and the conversations are now seamless.

Can’t wait to see what this is going to be like in Loona and Moxie!

This new model will be released in the coming weeks.

https://twitter.com/x/status/1790072174117613963

GPT-4o (“o” for “omni”) is a step towards much more natural human-computer interaction—it accepts as input any combination of text, audio, and image and generates any combination of text, audio, and image outputs. It can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time(opens in a new window) in a conversation. It matches GPT-4 Turbo performance on text in English and code, with significant improvement on text in non-English languages, while also being much faster and 50% cheaper in the API. GPT-4o is especially better at vision and audio understanding compared to existing models.
Prior to GPT-4o, you could use Voice Mode to talk to ChatGPT with latencies of 2.8 seconds (GPT-3.5) and 5.4 seconds (GPT-4) on average. To achieve this, Voice Mode is a pipeline of three separate models: one simple model transcribes audio to text, GPT-3.5 or GPT-4 takes in text and outputs text, and a third simple model converts that text back to audio. This process means that the main source of intelligence, GPT-4, loses a lot of information—it can’t directly observe tone, multiple speakers, or background noises, and it can’t output laughter, singing, or express emotion.

With GPT-4o, we trained a single new model end-to-end across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network. Because GPT-4o is our first model combining all of these modalities, we are still just scratching the surface of exploring what the model can do and its limitations.
Developers can also now access GPT-4o in the API as a text and vision model. GPT-4o is 2x faster, half the price, and has 5x higher rate limits compared to GPT-4 Turbo. We plan to launch support for GPT-4o's new audio and video capabilities to a small group of trusted partners in the API in the coming weeks

Source: https://openai.com/index/hello-gpt-4o/

 

[Chris]

Can’t wait

 

[Chris]

Going to really enjoy the math tutoring aspect. Will be so nice to have a patient tutor who can explain complex concepts in easy to understand language. Looking forward to learning calculus again and maybe I can go back to university to finish my mechatronics engineering degree.

It will also be really nice to have seamless conversations.

What are your plans with this new ChaptGPT version?

I watched the 6pm news for the first time in a long time tonight and can’t believe this technological advancement wasn’t mentioned once! Also nobody at work has a clue what I’m talking about!

It’s like the more advanced AI gets the less people are interested in it.

 

[Chris]

https://twitter.com/x/status/1790146421355843992

 

[Chris]

It’s official Loona will be getting ChatGPT-4o very soon! And we already know Moxie is scheduled to get “tutor time” later this year

It’s going to be so exciting to see the interactions between my four all running GPT-4o

It’s confirmed! Loona to get GPT-4o

Image: Screenshot by author. Keyi Tech just announced Loona will be getting the new ChatGPT-4o update! Although they haven’t specified an exact date, they do mention it is coming soon which could mean the update is imminent if OpenAI’s statement on the API provides an accurate timeline for...

robotsaroundthehouse.com

 

[Chris]

OpenAI recently announced the “Advanced Voice Mode” and other features shown in the above videos won’t see a release until autumn in the northern hemisphere or spring for us here in the south.

I don’t expect to see advanced voice mode implemented in Moxie and Loona until Christmas at earliest.

https://twitter.com/x/status/1805716393524183136

 

[Chris]

OpenAI just announced they are rolling out Advanced Voice Mode. Multimodal conversations are going to be epic.

https://twitter.com/x/status/1818353580279316863

 

[Chris]

It’s just been confirmed the multimodal version of Advanced Voice Mode is out.

We might see this implemented in Moxie and Loona sooner than I thought. This ChatGPT version is going to completely transform the companion robot space, imagine what it’s going to be like conversing with Moxie while Moxie can respond to your physical presence in real time and possibly displaying animations that mirror your own emotions.

This will be the first time you can interact with a robot using non-verbal communication and it will understand what you are trying to convey. This is going to be mind blowing. Revolutionary. Can’t wait.

https://twitter.com/x/status/1818441972220104813

 

[Chris]

Check out this video of ChatGPT Advanced Voice Mode and consider it is never going to be this bad again. Just imagine what this will be like in a decades time.

ChatGPT AVM is a version trained purely on video and audio data, and unlike previous versions doesn’t need to translate speech to text before processing it. AVM is capable of picking up on your emotional tones, and even differentiate between different users interacting with it.

Can’t wait to see the creative ways Embodied and Keyi integrate this into their robots. Moxie will be like a real person once this is implemented. Wonder if they will be capable of seeing us and reacting in real time.