The researchers trained a robot “chef” to watch and learn how to prepare videos, as well as recreate the dish.
Scientists at the University of Cambridge programmed their robot chef with a “cookbook” of eight simple salad recipes. After watching a video of a human demonstrating one of the recipes, the robot was able to identify which recipe was being prepared and cook it.
In addition, the videos helped the robot gradually add to its cookbook. At the end of the experiment, the robot came up with a ninth recipe on its own. Their results are reported in the journal IEEE Accessshow how video content can be a valuable and rich source of data for automated food production and can enable easier and cheaper deployment of robot chefs.
Robot chefs have been featured in sci-fi movies for decades, but in reality, cooking is a difficult task for a robot. Several commercial companies have developed prototype robot chefs, although none are currently commercially available, and they fall well short of their human counterparts in terms of skill.
Human chefs can learn new recipes through observation, whether it’s watching another person cook or watching a YouTube video, but programming a robot to cook a variety of dishes is expensive and time-consuming.
“We wanted to see if we could teach a robot chef to learn in the same incremental way that humans can, by figuring out the ingredients and how they go together in a dish,” said Grzegorz Soczacki, from Cambridge’s Department of Engineering, who first reported on the paper. : author:
Sochatsky, a PhD candidate in Professor Fumiya Iida’s Bio-Inspired Robotics Lab, and his colleagues developed 8 simple salad recipes and filmed them being made. They then used a publicly available neural network to train their robot chef. The neural network had already been programmed to identify a number of different objects, including fruits and vegetables used in eight salad recipes (broccoli, carrots, apples, bananas and oranges).
Using computer vision techniques, the robot analyzed each frame of the video and was able to recognize various objects and features, such as the knife and ingredients, as well as the hands, arms and face of the protester. Both the recipes and the videos were transformed into a vector, and the robot performed mathematical operations on the vectors to determine the similarity between the display and the vector.
By correctly identifying the ingredients and the human chef’s actions, the robot could determine which of the recipes was being prepared. The robot could deduce that if the protester held a knife in one hand and a carrot in the other, the carrot would then be cut.
Of the 16 videos it watched, the robot recognized the correct recipe 93% of the time, even though it only detected 83% of the human chef’s actions. The robot was also able to detect that minor variations in a recipe, such as making a double batch or common human error, were variations and not a new recipe. The robot also correctly recognized the demonstration of the new ninth salad, added it to its cookbook, and prepared it.
“It is surprising how many nuances the robot was able to detect,” said Sochaki. “These recipes are not complicated. they are essentially chopped fruits and vegetables, but it was really effective, for example, in understanding that two chopped apples and two chopped carrots are the same recipe as three chopped apples and three chopped carrots. “
The videos used to train the robot chef are not unlike the food videos made by some social media influencers, which are full of quick cuts and visual effects and move quickly between the cook and the dish being prepared. . For example, a robot would struggle to recognize a carrot if a human demonstrator wrapped its arm around it; for the robot to recognize the carrot, the human demonstrator had to hold the carrot up so the robot could see the whole vegetable.
“Our robot doesn’t care about food videos that go viral on social media, they’re just too hard to follow,” says Sochatsky. “But as these robot chefs get better and faster at identifying ingredients in food videos, they can use sites like YouTube to learn a whole range of recipes.”
The research was supported in part by Beko plc and the Engineering and Physical Sciences Research Council (EPSRC), part of UK Research and Innovation (UKRI).