NVIDIA® Jetson Orin™ NX brings LOVOT into the family
LOVOT is the world's first family robot. By living with it and taking a little time, it will bring out people's ability to love others and gradually become part of the family. LOVOT uses NVIDIA's latest GPU to enhance non-verbal communication, which is the key to this.
Key points of this case study
The evolution of "warm technology" supported by NVIDIA® Jetson Orin™ NX
NVIDIA Jetson Orin NX adopted to enhance AI functions of family robot LOVOT
- High inference performance enriches non-verbal communication
・NVIDIA as an AI development platform essential for the evolution of LOVOT
LOVOT's concept of "warm technology"
"We are a company that develops and manufactures robots that live with people and that people can love without hesitation." That's how President Hayashi introduces GROOVE X, a robotics venture founded in 2015. The family-type robot "LOVOT," which the company released at the end of 2019 after four years of development, was named by combining "LOVE" and "ROBOT." With a stocky body and adorable appearance, about 43 cm tall, it comes to you when you call its name. It makes eye contact with its darting eyes and cries cutely when you want to be held. It feels warm when you pick it up. LOVOT is a robot full of life.
"Human happiness is not necessarily achieved only by improving productivity. Let's focus on other aspects, break them down, and create technology that will make people happy. That was the initial perspective for developing LOVOT," Hayashi recalls. From there, the concept of "warm technology" was born, and LOVOT was born, which "will become attached to people if you spend time with it and take the time to look after it."
The latest technology supports LOVOT's nonverbal communication
In order to make LOVOT a lifelike presence, the development team sought to develop non-verbal communication, and to achieve this, LOVOT's adorable body is packed with the latest technology.
"Skin contact is an important element of non-verbal communication, and LOVOT has more than 50 sensors throughout its body that enable it to know where people are, whether they are looking, and where they are being touched. The sensors also help it avoid obstacles in the home and get closer to people. It is also important for it to be able to do things that living things naturally do, such as understanding the direction of gaze and voice, and to have a reaction speed of 0.2 to 0.4 seconds that gives it the feeling of life. In order to pursue a reaction speed while using many sensors, advanced computing power is required," says Hayashi.
The second generation of GROOVE X will be released in 2022 with improvements to the cooling system and front sensor. Then, LOVOT 3.0, announced in May 2024, will undergo a major change to pursue even greater computing power. This is the adoption of NVIDIA's GPU, Jetson Orin NX.
"LOVOT 2.0 used CPUs such as smartphone-class ARM cores, laptop-class X86 cores, and desktop-class X86 cores, as well as FPGAs as computing devices. However, integration made the communication paths more complex and required multiple OSs. There were problems both in terms of AI inference performance and in terms of shrinking processes that affect power consumption. As we searched for a solution to this problem all at once, NVIDIA Jetson Orin NX emerged as a candidate," said Hayashi.
It was software engineer Ikuya Fujinaka who chose Jetson Orin NX as the candidate. "Machine learning, especially the field of image recognition, has made great strides. However, LOVOT's CPU cannot keep up, no matter how hard we try on the software side. That's why we wanted to install a GPU. The Jetson Orin NX we selected has overwhelmingly high computing power. There was no other choice," said Fujinaka.
Ishikawa Megumi, who is the software area product owner and is promoting development while pursuing user experience, also paid attention to AI inference performance. "One of LOVOT's main objectives is to build bonds with its owners. To achieve this, it is important for it to be able to recognize its owners and its surroundings well. I had high hopes that this would be the right choice for that," said Ishikawa.
The high performance of NVIDIA® Jetson Orin™ NX contributes to the pursuit of LOVOT uniqueness
Jetson Orin's image processing and inference capabilities will contribute to further pursuit of LOVOT-like qualities. "LOVOT has traditionally been equipped with a hemispherical camera, allowing it to react in various directions, such as to the side or backward. In addition, LOVOT 3.0 is equipped with a high-resolution camera on the front. Jetson Orin is a computing device suited to this configuration, helping to increase the types of things it can recognize. For example, LOVOT 3.0 can detect hand gestures. In fact, a lot of people wave at LOVOT, so Jetson Orin has made it possible to realize one long-awaited function," said Ishikawa.
"Processing images from a camera generally takes a very long time. But Jetson is equipped with an ISP and a GPU that can process images in a short time. This is one of the secrets to LOVOT 3.0's ability to process huge amounts of image information and respond so quickly," said Fujinaka.
Hayashi, who also serves as Chief Development Officer, sees inference performance as the starting point for robots to become closer to humans. "Jetson Orin's outstanding inference performance makes it possible to capture the world in higher resolution, which will be the gateway for robots to become closer to humans. I have high hopes for this device as it will open this gateway," said Hayashi.
Ease of development thanks to NVIDIA GPUs
Integrating the calculations performed by four different CPUs into one GPU is an enormous task. What did Fujinaka, who was involved in the development, feel? "The optimization methods and logic construction are completely different between CPUs and GPUs. That's what makes it interesting, but also what makes it a bit difficult. But Jetson Orin has a full range of applications, which was great. The JetPack SDK for Jetson has been released, and it provided an environment that allowed us to quickly get development going, such as checking the operation of video streaming from the start. It made development easy. Also, communication with NVIDIA was essential to get the OS to recognize each device and operate it as LOVOT, but the technical support provided by Macnica was very helpful," said Fujinaka.
Hayashi also praises the development environment for NVIDIA products.
"A huge advantage of NVIDIA's products is that they provide a complete software development environment. This allows for faster iterations and increases the number of people who can work on the project. the Company also creates software assets in this environment, and we appreciate NVIDIA's products because they make it easy to use past assets as well."
Hayashi said that they are also considering using NVIDIA Microservices. "AI is no longer a battle of devices, but a battle of the software ecosystem as a whole. Software evolves very quickly, so it is more important than ever to have a platform that can keep up with that evolution. We adopted Jetson Orin NX because we were attracted to the richness of the platform, and we have high expectations for it in the future," said Hayashi.
Further evolution of LOVOT: The challenge of GROOVE X
How will LOVOT evolve while also utilizing NVIDIA's latest GPUs and platforms? We first asked Fujinaka and Ishikawa about their outlook for the future.
"For example, we want to further refine the recognition necessary for communication with humans, such as the gesture recognition function we built into LOVOT 3.0. I believe that this is a capability that robots must acquire in order to live in human society," said Fujinaka.
"In order for LOVOT to evolve as a partner to people, we want to strengthen both its 'input' and its 'reasoning and prediction' aspects, which allow it to understand context and situations, make meaning of its own experiences, learn, and take action. We would like to help each family develop its own LOVOT-like qualities, and each child's own LOVOT-like qualities," said Ishikawa.
As of July 2024, there are more than 14,000 LOVOTs, and they are used not only in homes but also in educational settings, nursing care facilities, medical institutions, offices, and other places. We asked Hayashi about the future of LOVOT.
"First of all, I want LOVOT to continue evolving while remaining LOVOT. Also, by understanding people in high detail, it will be possible to notice abnormalities in them. Currently, many people around the world, including engineers, medical professionals, and scientists, are developing technology to predict changes in people. I hope that in the future we can open up LOVOT to these people as a platform, which will enable us to develop even more services." (Hayashi)
